Apparatus and methods for use with image-guided skeletal procedures

ABSTRACT

Apparatus and methods are described including acquiring 3D image data of a targeted skeletal portion within a body of a subject, and a 2D radiographic image of the targeted skeletal portion. A machine-learning engine is used to generate machine-learning data based on (i) the 3D image data of the targeted skeletal portion, (ii) a database of 2D projection images generated from the 3D image data, and (iii) respective values of one or more viewing parameters corresponding to each 2D projection image. A computer processor receives the machine-learning data, receives the 2D radiographic image of the targeted skeletal portion, and registers the 2D radiographic image to the 3D image data by using the machine-learning data to find a 2D projection from the 3D image data that matches the 2D radiographic image of the targeted skeletal portion. Other applications are also described.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is a Continuation of U.S. Ser. No. 16/629,449,filed Jan. 8, 2020, which is the US national stage application ofPCT/IL2018/050732, filed Jul. 5, 2018, which published as PCTPublication WO 2019/012520 to Tolkowsky et al., and which claims thepriority of the following applications:

-   -   U.S. 62/530,123 to Tolkowsky et al., filed Jul. 8, 2017,        entitled, “Apparatus and methods for use with image-guided        skeletal procedures,”    -   U.S. 62/556,436 to Tolkowsky et al., filed Sep. 10, 2017,        entitled, “Apparatus and methods for use with image-guided        skeletal procedures,”    -   U.S. 62/599,802 to Tolkowsky et al., filed Dec. 18, 2017,        entitled, “Apparatus and methods for use with image-guided        skeletal procedures,” and    -   U.S. 62/641,359 to Tolkowsky et al., filed Mar. 11, 2018,        entitled, “Apparatus and methods for use with image-guided        skeletal procedures.”

Each of the abovementioned applications is incorporated herein byreference.

FIELD OF EMBODIMENTS OF THE INVENTION

Some applications of the present invention generally relate to medicalapparatus and methods. Specifically, some applications of the presentinvention relate to apparatus and methods for use in procedures that areperformed on skeletal anatomy.

BACKGROUND

Approximately 5 million spine surgeries are performed annuallyworldwide. Traditional, manual surgery is known as freehand surgery.Typically, for such procedures, a 3D scan (e.g., a CT and/or MRI) scanis performed prior to surgery. A CT scan is typically performed for bonytissue (e.g., vertebra), and an MRI scan is typically performed for softtissue (e.g., discs).

Reference is made to FIG. 1A, which is a schematic illustration of atypical set up of an orthopedic operating room, for procedures that areperformed in a freehand manner. Typically, in freehand procedures,although the CT and/or MRI scan is examined by the surgeon whenpreparing for surgery, no use is made of the CT and/or MRI images duringsurgery, other than potentially as a general reference that may beviewed occasionally. Rather, the surgery is typically performed under 2Dx-ray image guidance, the 2D x-rays typically being acquired using anx-ray C-arm. FIG. 1A shows a surgeon 10 performing a procedure usingintraprocedural x-ray images that are acquired by a C-arm 34, anddisplayed on a display 12. Freehand surgery in which there issignificant use of x-rays is known as fluoroscopy-guided surgery. X-rayC-arms are ubiquitous, familiar to surgeons, useful for acquiringreal-time images, tool-neutral (i.e., there is no requirement to useonly specific orthopedic tools that are modified specifically forimaging by that x-ray C-arm), and relatively inexpensive. A growingproportion of spinal surgeries are performed using a minimally-invasivesurgery (also known as “MIS,” or in the case of spine surgery,minimally-invasive spine surgery, which is also known as “MISS”), or“mini-open” surgery. In contrast to open surgery, in which an incisionis typically made along the entire applicable segment of the spine uponwhich surgery is performed, in minimally-invasive surgery, very smallincisions are made at the insertion point of tools. In “mini-open”surgery, incisions are made that are smaller than in open surgery andlarger than in minimally-invasive surgery. Typically, the less invasivethe type of surgery that is performed, the greater the use of x-rayimaging for assisting the procedure as the anatomy being operated on maynot all be in the surgeon's direct line of sight. There is evidence thatless invasive procedures that are performed under fluoroscopic guidanceenable faster patient recovery compared with open procedures. However,the use of real-time fluoroscopic guidance typically exposes thepatient, as well as the surgeon and the support staff to a relativelylarge amount of radiation.

A minority of procedures are performed using Computer Aided Surgery(CAS) systems that provide “GPS-like” navigation and/or robotics. Suchsystems typically make use of CT and/or MRI images that are generatedbefore the patient is in the operating room, or when the patient iswithin the operating room, but typically before an intervention hascommenced. The CT and/or MRI images are registered to the patient'sbody, and, during surgery, tools are navigated upon the images, thetools being moved manually, robotically or both.

Typically, in CAS procedures, a uniquely-identifiable location sensor isattached to each tool that needs to be tracked by the CAS system. Eachtool is typically identified and calibrated at the beginning of theprocedure. In addition, a uniquely-identifiable reference sensor isattached, typically rigidly, to the organ. In the case of spinalsurgery, the reference sensor is typically drilled into, or fixatedonto, the sacrum or spine, and, if surgery is performed along a numberof vertebrae, the reference sensor is sometimes moved and drilled into adifferent portion of the spine, mid-surgery, in order to always besufficiently close to the surgical site. The images to be navigated upon(e.g., CT, MRI), which are acquired before the patient is in theoperating room, or when the patient is within the operating room, butbefore an intervention has commenced, are registered to the patient'sbody or a portion thereof. In order to register the images to thepatient's body, the current location of the patient's body is broughtinto the same reference frame of coordinates as the images using thereference sensor. The location sensors on the tools and the referencesensor on the patient's body are then tracked, typically continuously,in order to determine the locations of the tools relative to thepatient's body, and a symbolic representation of the tool is displayedupon the images that are navigated upon. Typically, the tool and thepatient's body are tracked in 5-6 degrees of freedom.

There are various techniques that are utilized for the tracking oftools, as well as applicable portions of the patient's body, andcorresponding location sensors are used for each technique. Onetechnique is infrared (“IR”) tracking, whereby an array of cameras trackactive IR lights on the tools and the patient's body, or an array ofbeams and cameras tracks passive IR reflectors on the tools and thepatient's body. In both categories of IR tracking, lines of sight mustbe maintained at all times between the tracker and the tools. Forexample, if the line of sight is blocked by the surgeon's hands, thiscan interfere with the tracking. Another technique is electromagnetic ormagnetic tracking, whereby a field generator tracks receivers, typicallycoils, on the tools and the patient's body. For those latter techniques,environmental interferences from other equipment must be avoided oraccounted for. In each of the techniques, the location sensors of thenavigation system are tracked using tracking components that would notbe present in the operating room in the absence of the navigation system(i.e., the location sensors do not simply rely upon imaging by imagingdevices that are typically used in an orthopedic operating room in theabsence of the navigation system).

A further technique that can be used with a robotically-driven tool isto start with the tool at a known starting point relative to thepatient's body, and to then record motion of the tool from the startingpoint. Alternatively, such tools can be tracked using theabove-described techniques.

Given the nature of CAS procedures, the equipment required for suchprocedures is typically more expensive than that of non-CAS procedures(non-CAS procedures including open procedures, mini-open procedures, orminimally-invasive procedures that are not computer aided with respectto the guidance of tools). Such procedures typically limit toolselection to those fitted with location sensors as described above, andtypically require such tools to be individually identified andcalibrated at the beginning of each surgery.

SUMMARY OF EMBODIMENTS

In accordance with some applications of the present invention, thefollowing steps are typically performed during procedures that areperformed on skeletal anatomy, using a system that includes a computerprocessor. Such procedures may include joint (e.g., shoulder, knee, hip,and/or ankle) replacement, joint repair, fracture repair (e.g., femur,tibia, and/or fibula), a procedure that is performed on a rib (e.g., ribremoval, or rib resection), and/or other interventions in which 3D imagedata are acquired prior to the intervention and 2D images are acquiredduring the intervention. For some applications, the steps are performedduring a procedure that is performed on one or more vertebrae of asubject's spine and/or on other spinal elements.

Typically, in a first step, targeted vertebra(e) are marked by anoperator, typically prior to the actual intervention, with respect to 3Dimage data (e.g., a 3D image, a 2D cross-section derived from 3D imagedata, and/or a 2D projection image derived from 3D image data) of thesubject's spine. For some applications, pre-intervention planning isperformed. For example, desired insertion points, incision areas, ortool trajectories may be planned and associated with the 3D image data.For some applications, in a second step, a radiopaque element, such asthe tip of a surgical tool or a radiopaque marker, is placed in avicinity of the subject, e.g., on the subject, underneath the subject,on the surgical table, or above the surgical table. Typically, in athird step, vertebrae of the spine are identified in order to verifythat the procedure is being performed with respect to the correctvertebra (a step which is known as “level verification”), usingradiographic images of the spine and the markers to facilitate theidentification. For some applications, in a fourth step, an incisionsite (in the case of minimally-invasive surgery), or a tool entry pointinto a vertebra (in the case of open surgery) is determined upon thepatient's body. In a fifth step, the first tool in the sequence of tools(which in the case of minimally-invasive or less-invasive surgery istypically a needle, e.g., a Jamshidi™ needle) is typically inserted intothe subject (e.g., in the subject's back) via the incision site or thetool entry point, and is slightly fixated in the vertebra. In the caseof more-invasive or open spinal surgery, such tool is typically apedicle finder (which may also be known as a pedicle marker).Optionally, such tool is attached to a holder mechanism that istypically fixed to the surgical table but may also be fixed to a surfaceother than the surgical table, e.g., another table in the operatingroom, a stationary or movable stand, or imaging equipment inside theoperating room. In a sixth step, two or more 2D radiographic images aretypically acquired from respective views that typically differ by atleast 10 degrees, e.g., at least 20 degrees (and further typically by 30degrees or more), and one of which is typically from the direction ofinsertion of the tool. Typically, generally-AP and generally-lateralimages are acquired. Alternatively or additionally, images fromdifferent views are acquired. Typically, in a seventh step, the computerprocessor registers the 3D image data to the 2D images.

Typically, 3D image data and 2D images of individual vertebrae areregistered to each other. Further typically, the 3D image data and 2Dimages are registered to each other by generating a plurality of 2Dprojections from the 3D image data, and identifying respective first andsecond 2D projections that match each of the 2D x-ray images of thevertebra, as described in further detail hereinbelow. Typically, firstand second 2D x-ray images of the vertebra are acquired using an x-rayimaging device that is unregistered with respect to the subject's body,or whose precise pose relative to the subject's body (and morespecifically the applicable portion thereof) when acquiring images isnot known or tracked, by (a) acquiring a first 2D x-ray image of thevertebra (and the tool positioned relative to the vertebra, or at leasta portion of the tool inserted into the vertebra) from a first view,while the x-ray imaging device is disposed at a first pose with respectto the subject's body, (b) moving the x-ray imaging device to a secondpose with respect to the subject's body, and (c) while the x-ray imagingdevice is at the second pose, acquiring a second 2D x-ray image of atleast the portion of the tool and the vertebra from a second view. Forsome applications, more than two 2D x-rays are acquired from respectivex-ray image views, and the 3D image data and 2D x-ray images aretypically all registered to each other by identifying a correspondingnumber of 2D projections of the 3D image data that match respective 2Dx-ray images.

For some applications, the “level verification” is performed usingregistration of the 2D x-ray images to the 3D image data. For example,the system may attempt to register each 2D x-ray with the targetedvertebra in the 3D image until a match is found. The targeted vertebramay now be marked in the 2D x-ray and can be seen with respect to aradiopaque element that is placed in the vicinity of the subject andappears in the same 2D x-ray. Additionally or alternatively, the systemmay take a plurality of 2D x-ray images, each one being of a differentsegment of the anatomy, e.g., skeletal portion of the body, e.g., spine,and register all of them to the 3D image data of the anatomy. Usingpost-registration correspondence of each 2D x-ray image to the 3D imagedata, the plurality of 2D x-ray images may be related to each other soas to create a combined 2D x-ray image of the anatomy.

For some applications, the computer processor acquires a 2D x-ray imageof a tool inside, or relative to, the vertebra from only a single x-rayimage view, and the 2D x-ray image is registered to the 3D image data bygenerating a plurality of 2D projections from the 3D image data, andidentifying a 2D projection that matches the 2D x-ray image of thevertebra. In response to registering the 2D x-ray image to the 3D imagedata, the computer processor drives a display to display a cross-sectionderived from the 3D image data at a current location of a tip of thetool, as identified from the 2D x-ray image, and optionally to show avertical line on the cross-sectional image indicating a line within thecross-sectional image somewhere along which the tip of the tool iscurrently disposed.

As described hereinabove, typically two or more 2D x-rays are acquiredfrom respective x-ray image views, and the 3D image data and 2D imagesare typically registered to each other by identifying a correspondingnumber of 2D projections of the 3D image data that match the respective2D x-ray images. Subsequent to the registration of the 3D image data tothe 2D x-ray images, additional features of the system are applied bythe computer processor. For example, the computer processor may drivethe display to display the anticipated (i.e., extrapolated) path of thetool with reference to a target location and/or with reference to adesired insertion vector. For some applications, the computer processorsimulates tool progress within a secondary 2D imaging view, based uponobserved progress of the tool in a primary 2D imaging view.Alternatively or additionally, the computer processor overlays an imageof the tool, a representation thereof, and/or a representation of thetool path, upon the 3D image data (e.g., a 3D image, a 2D cross-sectionderived from 3D image data, and/or a 2D projection image derived from 3Dimage data), the location of the tool or tool path having been derivedfrom current 2D images.

For some applications, when more than one tool appear in the 2D x-rays,the system uses registration of two 2D x-ray images to 3D image datacontaining a pre-planned insertion path for each of the tools toautomatically associate between (a) a tool in a first one of the 2Dx-ray images and (b) the same tool in a second one of the 2D x-rayimages.

As described hereinabove, for some applications, sets of markers areplaced on the subject, underneath the subject, on the surgical table, orabove the surgical table. Typically, the markers that are placed atrespective locations with respect to the subject are identifiable inx-ray images, in optical images, and physically to the human eye. Forexample, respective radiopaque alphanumeric characters, arrangements ofa discernible shape, or particular symbols, may be placed at respectivelocations. For some applications, markers placed at respective locationsare identifiable based upon other features, e.g., based upon thedispositions of the markers relative to other markers. Using aradiographic imaging device, a plurality of radiographic images of theset of radiopaque markers are acquired, respective images being ofrespective locations along at least a portion of the subject's spine andeach of the images including at least some of the radiopaque markers.Using the computer processor, locations of the radiopaque markers withinthe radiographic images are identified, by means of image processing. Atleast some of the radiographic images are combined with respect to oneanother based upon the identified locations of the radiopaque markerswithin the radiographic images. Typically, such combination of images issimilar to stitching of images. However, the images may not necessarilybe precisely stitched such as to stitch portions of the subject'sanatomy in adjacent images to one another. Rather, the images arecombined with sufficient accuracy to be able to determine a location ofthe given vertebra within the combined radiographic images. Also, theexact pose or spatial position of the imaging device (e.g., the x-rayc-arm) when acquiring any of the images, relative to the subject's body(and more specifically the applicable portion thereof), need not beknown or tracked. The computer processor thus automatically determines(or facilitates manual determination of) a location of a given vertebrawithin the combined radiographic images. Based upon the location of thegiven vertebra within the combined radiographic images, a location ofthe given vertebra in relation to the set of radiopaque markers that isplaced on the subject is determined, as described in further detailhereinbelow. The markers are typically utilized to provide additionalfunctionalities, or in some cases to facilitate functionalities, asdescribed in further detail hereinbelow.

There is therefore provided, in accordance with some applications of thepresent invention, a method for performing a procedure with respect to askeletal portion within a body of a subject, the method including:

acquiring 3D image data of at least the skeletal portion;

using at least one computer processor:

-   -   designating at least one point selected from the group        consisting of (a) a skin-level incision point corresponding to        the skeletal portion and (b) a skeletal-portion-level entry        point within the body of the subject, and    -   associating the designated point with the 3D image data for the        skeletal portion;

positioning a radiopaque element on the body of the subject with respectto the skeletal portion, the radiopaque element being visible to thenaked eye;

acquiring an intraoperative 2D radiographic image of the skeletalportion, such that the radiopaque element appears in the 2D radiographicimage;

using the at least one computer processor:

-   -   registering the 2D radiographic image to the 3D image data such        that the designated point appears on the 2D radiographic image,    -   displaying a location of the designated point with respect to        the radiopaque element on the 2D radiographic image.

For some applications, the skeletal portion is a vertebra of a spine ofa subject.

For some applications, associating the designated point with the 3Dimage data includes storing the designated point as 3D coordinateswithin the 3D image data.

For some applications:

positioning the radiopaque element includes positioning a set ofradiopaque markers on the body of the subject with respect to theskeletal portion, the radiopaque markers being visible to the naked eye,and

displaying the location of the designated point with respect to theradiopaque element includes displaying the location of the designatedpoint with respect to the set of radiopaque markers on the 2Dradiographic image.

For some applications, registering the 2D radiographic image to the 3Dimage data includes:

generating a plurality of 2D projections from the 3D image data; and

identifying a 2D projection that matches the 2D radiographic image ofthe skeletal portion.

For some applications, the method further includes, based on thelocation of the designated point with respect to the radiopaque elementon the 2D radiographic image, labeling a location of the designatedpoint on the subject's body.

There is further provided, in accordance with some applications of thepresent invention, apparatus for performing a procedure on a skeletalportion within a body of a subject, the apparatus comprising:

a radiopaque marker configured to be attached to a surface of thesubject in a vicinity of the skeletal portion such that the radiopaquemarker appears in radiographic images of the skeletal portion that areacquired from a first image view, the radiopaque marker including atleast one 2D foldable segment,

-   -   each 2D foldable segment configured to be foldable away from the        surface of the subject such that if folded the 2D foldable        segment appears in radiographic images of the skeletal portion        that are acquired from a second image view that is different        from the first image view,    -   the at least one 2D foldable segment thereby facilitating        associating a given skeletal portion that appears in the        radiographic images of the skeletal portion from the first image        view, with the given skeletal portion in the radiographic images        of the skeletal portion from the second image view, by        facilitating identification of the at least one 2D foldable        segment in the radiographic images of the skeletal portion        acquired from the second image view.

For some applications, the skeletal portion is a spine of a subject.

For some applications, the second image view is different from the firstimage view by at least 20 degrees.

For some applications, the first image view is an anteroposterior (AP)image view of the skeletal portion.

For some applications, the second image view is a lateral image view ofthe skeletal portion.

For some applications, a fold line of the 2D foldable segment ispre-designated.

For some applications, the apparatus further includes an adhesivedisposed on the radiopaque marker.

For some applications, no adhesive is disposed on any of the at leastone 2D foldable segments.

For some applications:

further comprising a support comprising a set of discretely identifiablesupport-affixed radiopaque markers, such that the set of discretelyidentifiable support-affixed radiopaque markers appear in radiographicimages of the skeletal portion that are acquired from the first imageview, and

the at least one 2D foldable segment comprises at least one 2D foldabletab coupled to the support, wherein each 2D foldable tab comprises atleast one tab-affixed radiopaque marker such that if the 2D foldable tabis folded away from the surface of the subject, the at least onetab-affixed radiopaque marker appears in radiographic images of theskeletal portion that are acquired from the second image view.

For some applications, the at least one 2D foldable segment isconfigured to be converted to a 3D element when folded away from thesurface of the subject, such that, if folded, the 3D element appears inradiographic images acquired from at least the first and second imageviews.

There is further provided, in accordance with some applications of thepresent invention, a method for performing a procedure on a skeletalportion within a body of a subject, the method including:

attaching a radiopaque marker to a surface of the subject in a vicinityof the skeletal portion, the radiopaque marker comprising at least one2D foldable segment;

acquiring a radiographic image, from a first image view, of (i) theskeletal portion and (ii) the radiopaque marker; and

acquiring a radiographic image of (i) the skeletal portion and (ii) theat least one 2D foldable segment from a second image view that isdifferent from the first image view when the 2D foldable segment isfolded away from the surface of the subject.

For some applications, attaching includes attaching the radiopaquemarker to a surface of the subject in the vicinity of a spine of thesubject.

For some applications, acquiring the radiographic image of (i) theskeletal portion and (ii) the at least one 2D foldable segment from thesecond image view includes acquiring the radiographic image of (i) theskeletal portion and (ii) the at least one 2D foldable segment from asecond image view that is different from the first image view by atleast 20 degrees.

For some applications, attaching includes attaching a radiopaque markerincluding a support,

(a) the support including a series of discretely identifiablesupport-affixed radiopaque markers, and acquiring the radiographicimage, from the first image view, of (i) the skeletal portion and (ii)the radiopaque marker including acquiring a radiographic image, from thefirst image view, of (i) the skeletal portion and (ii) the series ofdiscretely identifiable support-affixed radiopaque markers, and

(b) the at least one 2D foldable segment including at least one 2Dfoldable tab coupled to the support, each 2D foldable tab including atleast one tab-affixed radiopaque marker, and acquiring the radiographicimage of (i) the skeletal portion and (ii) the at least one 2D foldablesegment from a second image view including acquiring a radiographicimage of (i) the skeletal portion and (ii) the at least one tab-affixedradiopaque marker.

There is further provided, in accordance with some applications of thepresent invention, a method for performing a procedure with respect to atargeted vertebra of a spine within a body of a subject, the methodincluding:

acquiring 3D image data of at least the targeted vertebra;

using at least one computer processor, indicating the targeted vertebrawithin the 3D image data;

positioning a radiopaque element on the body of the subject with respectto the spine of the subject, the radiopaque element being visible to thenaked eye;

acquiring a 2D radiographic image of the spine of the subject, such thatthe radiopaque element appears in the radiographic image; and

using the computer processor, registering the targeted vertebra in the3D image data to the targeted vertebra in the 2D radiographic image, theregistering including:

-   -   generating a plurality of 2D projections of the targeted        vertebra from the 3D image data,    -   for each vertebra that is visible in the 2D radiographic image,        identifying if there exists a 2D projection of the targeted        vertebra that matches the 2D radiographic image of the vertebra        that is visible in the 2D radiographic image, and    -   in response to the identifying, indicating on the 2D        radiographic image the vertebra for which a match with a 2D        projection of the targeted vertebra was identified, such that a        location of the targeted vertebra is identified with respect to        the radiopaque element.

For some applications, the method further includes, based on theidentified location of the targeted vertebra with respect to theradiopaque element, positioning an intraoperative 3D imaging device suchthat an imaging volume of the 3D imaging device at least partiallyoverlaps the targeted vertebra.

For some applications, positioning the radiopaque element includespositioning at least one radiopaque marker on the body of the subjectwith respect to the spine of the subject, the at least one radiopaquemarker being visible to the naked eye.

For some applications, positioning the radiopaque element includespositioning a radiopaque surgical tool on the body of the subject withrespect to the spine of the subject.

There is further provided, in accordance with some applications of thepresent invention, a method for registering a 2D radiographic image of atargeted skeletal portion within a body of a subject to 3D image data ofthe targeted skeletal portion, the method including:

acquiring 3D image data of the targeted skeletal portion;

obtaining deep-learning data by inputting into a deep-learning engine(a) a database of 2D projection images generated from the 3D image data,and (b) respective values of viewing parameters corresponding to each 2Dprojection image, such that given a certain 2D projection image, thedeep-learning engine learns to suggest simulated respective values ofviewing parameters that correspond to that 2D projection image;

inputting the obtained deep-learning data into at least one computerprocessor;

acquiring an intraoperative 2D radiographic image of the targetedskeletal portion; and

registering the intraoperative 2D radiographic image of the targetedskeletal portion to the 3D image data of the targeted skeletal portion,the registering including:

-   -   using the computer processor:        -   using the obtained deep-learning data to limit a search            space in which a 2D projection from the 3D image data that            matches the 2D radiographic image of the targeted skeletal            portion should be searched for, and        -   searching for a 2D projection that matches the 2D            radiographic image of the targeted skeletal portion only            within the limited search space.

For some applications, acquiring the 3D image data includes acquiring 3Dimage data of a targeted vertebra of a spine of the subject.

For some applications, obtaining the deep learning data includesobtaining deep learning data by inputting into the deep-learning engine(a) a database of 2D projection images generated from the 3D image data,and (b) respective viewing distances and viewing angles corresponding toeach 2D projection image, such that given a certain 2D projection image,the deep learning engine learns to suggest a simulated respectiveviewing distance and viewing angle that correspond to that 2D projectionimage.

There is further provided, in accordance with some applications of thepresent invention, a method for use with at least two tools configuredto be advanced into a skeletal portion within a body of a subject alongrespective longitudinal insertion paths, the method including:

acquiring 3D image data of the skeletal portion;

planning the respective longitudinal insertion paths;

associating the planned respective longitudinal insertion paths with the3D image data;

while respective portions of the tools are disposed at first respectivelocations along their respective longitudinal insertion paths withrespect to the skeletal portion, sequentially:

-   -   acquiring a first 2D x-ray image of at least the respective        portions of the tools and the skeletal portion from a first        view, using a 2D x-ray imaging device that is disposed at a        first pose with respect to the subject's body;    -   moving the 2D x-ray imaging device to a second pose with respect        to the subject's body; and    -   while the 2D x-ray imaging device is at the second pose,        acquiring a second 2D x-ray image of at least the respective        portions of the tools and the skeletal portion from a second        view; and

using at least one computer processor, automatically matching between atool in the first 2D x-ray image and the same tool in the second 2Dx-ray image, the matching including:

-   -   (A) identifying respective tool elements of each of the tools        within each of the first and second 2D x-ray images, by means of        image processing;    -   (B) registering the first and second x-ray images to the 3D        image data, the registering including:        -   generating a plurality of 2D projections from the 3D image            data, and        -   identifying respective first and second ones of the 2D            projections that match the first and second 2D x-ray images            of the skeletal portion; and    -   (C) based upon the identified respective tool elements within        the first and second 2D x-ray images, and the registration of        the first and second 2D x-ray images to the 3D image data,        identifying for at least one tool element within the first and        second 2D x-ray images a correspondence between (i) the tool        element and (ii) the respective planned longitudinal insertion        path for that tool.

For some applications, associating the planned respective longitudinalinsertion paths with the 3D image data includes displaying each plannedlongitudinal insertion path distinctively within the 3D image data.

For some applications, the method further includes, using the at leastone computer processor, based on the identified respective tool elementswithin the first and second 2D x-ray images, and the registration of thefirst and second 2D x-ray images to the 3D image data, overlaying theplanned respective longitudinal insertion paths distinctively on thefirst and second 2D x-ray images.

For some applications, the method further includes, using the at leastone computer processor, based on the identified respective tool elementswithin the first and second 2D x-ray images, and the registration of thefirst and second 2D x-ray images to the 3D image data, positioningrespective representations of the respective tool elements within adisplay of the 3D image data.

For some applications, acquiring the first 2D x-ray image of at leastthe respective portions of the tools and the skeletal portion from thefirst view includes using a 2D x-ray imaging device that is unregisteredwith respect to the body of the subject.

There is further provided, in accordance with some applications of thepresent invention, a method for performing a procedure with respect to agiven vertebra of a spine within a body of a subject, the methodincluding:

placing a set of radiopaque markers in a vicinity of the subject, themarkers being visible to the naked eye;

using a radiographic imaging device, acquiring a plurality ofradiographic images of the set of radiopaque markers, respective imagesbeing of respective locations along at least a portion of the subject'sspine and each of the images including at least some of the radiopaquemarkers;

using at least one computer processor:

-   -   identifying locations of the radiopaque markers within the        radiographic images, by means of image processing;    -   combining at least some of the radiographic images based upon        the identified locations of the radiopaque markers within the        radiographic images;    -   determining a location of the given vertebra within the combined        radiographic images; and    -   generating an output in response thereto; and

based on the identified location of the given vertebra with respect tothe radiopaque markers within the combined radiographic images, manuallyidentifying a location of the given vertebra on the subject's body withrespect to the markers positioned in the vicinity of the subject; and

positioning an intraoperative 3D imaging device such that an imagingvolume of the 3D imaging device at least partially overlaps the givenvertebra to be subsequently operated on.

For some applications, placing the set of radiopaque markers in thevicinity of the subject includes placing the set of radiopaque markersin the vicinity of the subject such that the set of radiopaque markersis in contact with the subject.

For some applications, placing the set of radiopaque markers in thevicinity of the subject includes placing the set of radiopaque markersin the vicinity of the subject such that the set of radiopaque markersis not in contact with the subject.

For some applications, determining the location of the given vertebrawithin the combined radiographic images includes, using the at least onecomputer processor, determining the location of the given vertebrawithin the combined radiographic images by means of image processing.

There is further provided, in accordance with some applications of thepresent invention, a method for performing a procedure using a toolconfigured to be advanced into a skeletal portion within a body of asubject along a longitudinal insertion path, the method including:

acquiring 3D image data of the skeletal portion;

while a portion of the tool is disposed at a first location along thelongitudinal insertion path with respect to the skeletal portion,sequentially:

-   -   1. acquiring a first 2D x-ray image of at least the portion of        the tool and the skeletal portion from a first view, using a 2D        x-ray imaging device that is disposed at a first pose with        respect to the subject's body;    -   2. moving the 2D x-ray imaging device to a second pose with        respect to the subject's body; and    -   3. while the 2D x-ray imaging device is at the second pose,        acquiring a second 2D x-ray image of at least the portion of the        tool and the skeletal portion from a second view;

using at least one computer processor:

-   -   registering the first and second 2D radiographic images to the        3D image data, the registering including:        -   generating a plurality of 2D projections from the 3D image            data, and        -   identifying respective first and second 2D projections that            match the first and second 2D radiographic images of the            skeletal portion;    -   identifying a location of the portion of the tool with respect        to the skeletal portion, within the first and second 2D        radiographic images, by means of image processing; and    -   based upon (a) the identified location of the portion of the        tool within the first and second 2D x-ray images, and (b) the        registration of the first and second 2D x-ray images to the 3D        image data, identifying the first location of the portion of the        tool with respect to the 3D image data;

computing an anticipated longitudinal forward path of the tool withinthe 3D image data from the first and second 2D radiographic images;

subsequently, moving the portion of the tool to a second location alongthe longitudinal insertion path with respect to the skeletal portion;

subsequent to moving the portion of the tool to the second location,acquiring one or more additional 2D radiographic images of at least theportion of the tool and the skeletal portion from a single image view;and

using the computer processor, facilitating identifying whether or notthe tool has deviated from the anticipated longitudinal forward path by:

-   -   registering the additional one or more 2D radiographic images to        the 3D image data, such that the anticipated longitudinal        forward path of the tool is registered with the additional one        or more 2D radiographic images, and    -   identifying a location of the portion of the tool with respect        to the skeletal portion, within the additional one or more 2D        radiographic images, by means of image processing.

For some applications, acquiring the first 2D x-ray image of at leastthe portion of the tool and the skeletal portion from the first viewincludes using a 2D x-ray imaging device that is unregistered withrespect to the body of the subject.

For some applications, the method further includes, using the computerprocessor, subsequently to registering the additional one or more 2Dradiographic images to the 3D image data, overlaying the anticipatedlongitudinal forward path of the tool on the additional one or more 2Dradiographic images.

There is further provided, in accordance with some applications of thepresent invention, a method for registering a 2D radiographic image of atargeted skeletal portion within a body of a subject to 3D image data ofthe targeted skeletal portion, the method including:

during a pre-processing phase:

-   -   acquiring 3D image data of the skeletal portion; and    -   using at least one computer processor:        -   generating N 2D projection images from the 3D image data,        -   determining a set of attributes that describe each of the 2D            projection images, the number of the attributes being            smaller than a number of pixels in each 2D projection image,        -   determining, for each 2D projection image, a respective            value for each of the attributes, and        -   storing N respective sets of attributes, with respective            values assigned for each attribute, for the N 2D projection            images;

following the pre-processing phase, during a medical procedure:

-   -   acquiring a 2D radiographic image of the skeletal portion; and    -   using at least one computer processor:        -   determining at least one specific set of values for the            attributes that describe at least a portion of the 2D            radiographic image,        -   searching among the stored N respective sets of attributes            for a set that best matches any of the at least one specific            set of values, and        -   using the set that best matches, generating an additional 2D            projection image from the 3D image data, the additional 2D            projection image matching at least the portion of the 2D            radiographic image.

For some applications, the method further includes discarding the N 2Dprojection images subsequently to storing the N respective sets ofattributes.

There is further provided, in accordance with some applications of thepresent invention, a method for registering a 2D radiographic image of atargeted skeletal portion within a body of a subject to 3D image data ofthe targeted skeletal portion, the method including:

during a pre-processing phase:

-   -   acquiring 3D image data of the skeletal portion;    -   and using at least one computer processor:        -   generating N 2D projection images from the 3D image data,        -   determining a set of attributes that describe each of the 2D            projection images, the number of the attributes being            smaller than a number of pixels in each 2D projection image,        -   determining, for each 2D projection image, a respective            value for each of the attributes, and        -   storing N respective sets of attributes, with respective            values assigned for each attribute, for the N 2D projection            images;

following the pre-processing phase, during a medical procedure:

-   -   acquiring a 2D radiographic image of the skeletal portion; and    -   using at least one computer processor:        -   determining at least one specific set of values for the            attributes that describe at least a portion of the 2D            radiographic image,        -   searching among the stored N respective sets of attributes            for a set that best matches any of the at least one specific            set of values, and        -   using the set that best matches, generating a plurality of            additional 2D projection images from the 3D image data, each            of the plurality of additional projection images            approximating at least the portion of the 2D radiographic            image, and        -   using the plurality of additional projection images,            optimizing to find a 2D projection image that matches at            least the portion of the 2D radiographic image.

For some applications, the method further includes discarding the N 2Dprojection images subsequently to storing the N respective sets ofattributes.

There is further provided, in accordance with some applications of thepresent invention, a method for performing a procedure with respect to askeletal portion with the body of a subject, the method including:

acquiring 3D image data of the skeletal portion;

acquiring a plurality of 2D radiographic images, each image showing adistinct segment of the skeletal portion;

using at least one computer processor:

-   -   registering the 2D radiographic images with the 3D image data,        such that a post-registration correspondence is created between        each 2D radiographic image and the 3D image data;    -   using the post-registration correspondence between each of the        2D radiographic images and the 3D image data, relating the 2D        images with respect to each other; and    -   using the relationship of the 2D radiographic images with        respect to each other, generating a combined 2D radiographic        image comprising multiple segments of the skeletal portion.

For some applications, acquiring the plurality of 2D radiographic imagesincludes acquiring the plurality of 2D radiographic images from asimilar viewing direction.

For some applications, acquiring the plurality of 2D radiographic imagesincludes acquiring at least two of the 2D radiographic images fromviewing directions that are not similar to one another.

For some applications, acquiring the plurality of 2D radiographic imagesincludes acquiring the plurality of 2D radiographic images such thatthere is overlap between at least two of the segments shown in tworespective 2D radiographic images.

For some applications, acquiring the plurality of 2D radiographic imagesincludes acquiring the plurality of 2D radiographic images such that atleast two of the segments shown in two respective 2D radiographic imagesdo not overlap with each other.

For some applications, acquiring 3D image data of the skeletal portionincludes acquiring 3D image data of a spine of the subject.

For some applications, generating a combined 2D radiographic imageincludes multiple segments of the skeletal portion comprises generatinga combined 2D radiographic image comprising multiple segments of thespine, and the method further comprising, using the combinedradiographic image of the spine, identifying a given vertebra of thespine of the subject.

The present invention will be more fully understood from the followingdetailed description of embodiments thereof, taken together with thedrawings, in which:

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A is a schematic illustration of an orthopedic operating room, asused in prior art techniques;

FIG. 1B is a schematic illustration of a system for use with proceduresthat are performed on skeletal anatomy, in accordance with someapplications of the present invention;

FIG. 2 is a schematic illustration of two tools (e.g., Jamshidi™needles) being inserted into a vertebra and the desired insertionwindows for the insertion of such tools, as used in prior arttechniques;

FIGS. 3A and 3B show a 3D CT image of a vertebra (FIG. 3A), as well as a2D axial slice that is derived from the 3D CT image (FIG. 3B), as usedin prior art techniques;

FIGS. 4A and 4B are schematic illustrations showing a C-arm being usedto acquire an anteroposterior (“AP”) 2D radiographic image and aresultant AP image (FIG. 4A), and the C-arm being used to acquire alateral 2D radiographic image and a resultant lateral image (FIG. 4B),as used in prior art techniques;

FIGS. 5A, 5B, 5C, and 5D are schematic illustrations of radiopaquemarkers which are placed upon a subject and include at least one 2Dfoldable segment, in accordance with some applications of the presentinvention;

FIG. 5E is a flowchart showing steps that are typically performed usinga set of radiopaque markers, in accordance with some applications of thepresent invention;

FIG. 6 is a flowchart showing steps that are typically performed usingthe system of FIG. 1B, in accordance with some applications of thepresent invention;

FIG. 7 shows a vertebra designated upon cross-sectional images of asubject's spine that are derived from 3D image data, in accordance withsome applications of the present invention;

FIG. 8A is a flow chart showing a method for level verification with anadditional step of positioning an intraoperative 3D imaging device, inaccordance with some applications of the present invention;

FIG. 8B shows an example of a 3D CT image of a subject's spine displayedalongside a combined radiographic image of the subject's spine, inaccordance with some applications of the present invention;

FIG. 8C shows the designated vertebra indicated on a 3D CT image and ona 2D x-ray image, the CT image and x-ray image being displayed alongsideone another, in accordance with some applications of the presentinvention;

FIGS. 8D-8I, show an example of generating a combined spinal image fromfour individual x-ray images that were acquired sequentially along thespine, in accordance with some applications of the present invention;

FIGS. 9A and 9B are flow charts showing another method for performinglevel verification, in accordance with some applications of the presentinvention;

FIG. 10 shows an example of an optical image displayed alongside a 2Dradiographic image, in accordance with some applications of the presentinvention;

FIG. 11 shows an example of a 2D radiographic (e.g., x-ray) imagedisplayed alongside a cross-sectional image of a subject's vertebra thatis derived from 3D image data of the vertebra, in accordance with someapplications of the present invention;

FIG. 12A shows an example of an identification upon the subject of apreviously-planned skin-level incision point, in accordance with someapplications of the present invention;

FIG. 12B is a flowchart of a method for determining a designated, e.g.,planned, point for skin-level or skeletal-portion-level incision/entry,in accordance with some applications of the present invention;

FIGS. 12C-12J show a method for determining a designated, e.g., planned,point for skin-level or skeletal-portion-level incision/entry, inaccordance with some applications of the present invention;

FIGS. 12K-P show a method for determining a designated, e.g., planned,point for skin-level or skeletal-portion-level incision/entry, inaccordance with some applications of the present invention;

FIGS. 13A and 13B show an example of planning incision or tool insertionsites upon 3D scan data of a skeletal portion of the body, in accordancewith some applications of the present invention;

FIGS. 14A and 14B show examples of respectively AP and lateral x-rayimages of a Jamshidi™ needle being inserted into a subject's spine, inaccordance with some applications of the present invention;

FIGS. 15A and 15B show examples of correspondence between respectiveviews of a 3D image of a vertebra, with corresponding respective firstand second x-ray images of the vertebra, in accordance with someapplications of the present invention;

FIG. 16 is a schematic illustration showing tool gripped by anadjustable tool holder, with the tool holder fixed to the rail of asurgical table, in accordance with some applications of the presentinvention;

FIGS. 17A, 17B, and 17C are schematic illustrations that demonstrate therelationship between a 3D image of an object (FIG. 17A) and side-to-side(FIG. 17B) and bottom-to-top (FIG. 17C) 2D projection images of theobject, such a relationship being utilized, in accordance with someapplications of the present invention;

FIG. 18 is a flowchart showing steps that are performed by computerprocessor, in order to register 3D image data of a vertebra to two ormore 2D x-ray images of the vertebra, in accordance with someapplications of the present invention;

FIG. 19A is a flowchart showing steps of an algorithm that is performedby a computer processor, in accordance with some applications of thepresent invention;

FIGS. 19B-19E show an example of the automatic detection within an x-rayimage of a tool that is inserted into a vertebra, in accordance withsome applications of the present invention;

FIG. 20A shows an example of axial cross-sections of a vertebracorresponding, respectively, to first and second locations of a tip of atool that is advanced into the vertebra along a longitudinal insertionpath, as shown on corresponding 2D x-ray images that are acquired from asingle x-ray image view, in accordance with some applications of thepresent invention;

FIG. 20B shows an example of axial cross-sections of a vertebra uponwhich, respectively, first and second locations of a tip of a tool thatis advanced into the vertebra along a longitudinal insertion path aredisplayed, the locations being derived using x-ray images acquired fromtwo or more x-ray image views, in accordance with some applications ofthe present invention;

FIGS. 21A and 21B show examples of a display showing a given locationdesignated upon 3D (e.g., CT or MRI) image data and a relationshipbetween an anticipated longitudinal insertion path of a tool and thegiven location upon, respectively, AP and lateral 2D x-ray images, inaccordance with some applications of the present invention;

FIG. 21C shows an example of the representations of a portion of anactual tool and the planned insertion path displayed together within asemi-transparent 3D model of a spinal segment, in accordance with someapplications of the present invention;

FIG. 22A shows an AP x-ray of two tools being inserted into a vertebrathrough, respectively, 10-11 o'clock and 1-2 o'clock insertion windows,the AP x-ray being generated using prior art techniques;

FIG. 22B shows a corresponding lateral x-ray image to FIG. 17A, thelateral x-ray being generated using prior art techniques;

FIGS. 23A and 23B are flowcharts for a method for matching between atool in one x-ray image acquired from a first view, and the same tool ina second x-ray image acquired from a second view, in accordance withsome applications of the present invention;

FIG. 24 is a schematic illustration of a Jamshidi™ needle with aradiopaque clip attached thereto, in accordance with some applicationsof the present invention;

FIG. 25A shows an AP x-ray image and a corresponding lateral x-ray imageof a vertebra, at a first stage of the insertion of a tool into thevertebra, in accordance with some applications of the present invention;

FIG. 25B shows an AP x-ray image of the vertebra, at a second stage ofthe insertion of the tool into the vertebra, and an indication of thederived current location of the tool tip displayed upon a lateral x-rayimage of the vertebra, in accordance with some applications of thepresent invention;

FIG. 26 is a schematic illustration of a three-dimensional rigid jigthat comprises at least portions thereof that are radiopaque andfunction as radiopaque markers, the radiopaque markers being disposed ina predefined three-dimensional arrangement, in accordance with someapplications of the present invention;

FIGS. 27A and 27B are flowcharts for a method for verifying if the toolhas indeed proceeded along an anticipated longitudinal path, inaccordance with some applications of the present invention;

FIGS. 28A-E show an example of a tool bending during its insertion, withthe bending becoming increasingly visible (manually) or identifiable(automatically), in accordance with some applications of the presentinvention;

FIG. 29A show examples of x-ray images of a calibration jig generated bya C-arm that uses an image intensifier, and by a C-arm that uses aflat-panel detector, such images reflecting prior art techniques;

FIG. 29B shows an example of an x-ray image acquired by a C-arm thatuses an image intensifier, the image including a radiopaque componentthat corresponds to a portion of a tool that is known to be straight,and a dotted line overlaid upon the image indicating how a line definedby the straight portion would appear if distortions in the image arecorrected, in accordance with some applications of the presentinvention;

FIG. 30 is a flow chart for a method for dividing the registration intothe pre-processing phase and online phase, i.e., during a medicalprocedure, in accordance with some applications of the presentinvention;

FIG. 31 is a flow chart for a method for dividing the registration intothe pre-processing phase and online phase, i.e., during a medicalprocedure, in accordance with some applications of the presentinvention; and

FIG. 32 is a flow chart showing a method for image stitching, inaccordance with some applications of the present invention.

DETAILED DESCRIPTION OF EMBODIMENTS

Reference is now made to FIG. 1B, which is a schematic illustration of asystem 20 for use with procedures that are performed on skeletalanatomy, in accordance with some applications of the present invention.For some applications, the system is used for a procedure that isperformed on one or more vertebrae, or other portions of the spine.However, the scope of the present invention includes applying any of theapparatus and methods described herein to procedures performed on otherportions of a subject's skeletal anatomy, mutatis mutandis. Suchprocedures may include joint (e.g., shoulder, elbow, wrist, knee, hip,and/or ankle) replacement, joint repair, fracture repair (e.g., femur,tibia, and/or fibula), a procedure that is performed on a rib (e.g., ribremoval, or rib resection), and/or other interventions in which 3D imagedata are acquired prior to the intervention and 2D images are acquiredduring the intervention.

System 20 typically includes a computer processor 22, which interactswith a memory 24, and one or more user interface device 26. Typically,the user interface devices include one or more input devices, such as akeyboard 28 (as shown), and one or more output devices, e.g., a display30, as shown. Inputs to, and outputs from, the computer processor thatare described herein are typically performed via the user interfacedevices. For some applications, the computer processor as well as thememory and the user interface devices, are incorporated into a singleunit, e.g., a tablet device, an all-in-one computer, and/or a laptopcomputer.

For some applications, the user interface devices include a mouse, ajoystick, a touchscreen device (such as a smartphone or a tabletcomputer) optionally coupled with a stylus, a touchpad, a trackball, avoice-command interface, a hand-motion interface, and/or other types ofuser interfaces that are known in the art. For some applications, theoutput device includes a head-up display and/or a head-mounted display,such as Google Glass® or a Microsoft HoloLens®. For some applications,the computer processor generates an output on a different type ofvisual, text, graphics, tactile, audio, and/or video output device,e.g., speakers, headphones, a smartphone, or a tablet computer. For someapplications, a user interface device acts as both an input device andan output device. For some applications, computer processor 22 generatesan output on a computer-readable medium (e.g., a non-transitorycomputer-readable medium), such as a disk or a portable USB drive. Forsome applications, the computer processor comprises a portion of apicture archiving and communication system (PACS), and is configured toreceive inputs from other components of the system, e.g., via memory 24.Alternatively or additionally, the computer processor is configured toreceive an input on a computer-readable medium (e.g., a non-transitorycomputer-readable medium), such as a disk or a portable USB drive. It isnoted that, for some applications, more than one computer processor isused to perform the functions described herein as being performed bycomputer processor 22.

Typically, 3D image data are acquired before the subject is in theoperating room for the procedure, or when the subject is in theoperating room, but before an intervention has commenced. For example,3D CT image data of the portion of the skeletal anatomy upon which theprocedure is to be performed (and/or neighboring portions of theanatomy) may be acquired using a CT scanner 32. Alternatively oradditionally, 3D MRI image data of the portion of the skeletal anatomyupon which the procedure is to be performed (and/or neighboring portionsof the anatomy) may be acquired using an MRI scanner. For someapplications, 3D x-ray data are acquired. Typically, the 3D image dataare transferred to memory 24, and are retrieved from the memory bycomputer processor 22. It is noted that for illustrative purposes, FIG.1B shows the CT scanner, the C-arm, and system 20 together with oneanother. However, in accordance with the above description, for someapplications, the CT scanner is not disposed in the same room as system20, and/or C-arm 34.

During the procedure, real time 2D images are acquired by a radiographicimaging device, e.g., a C-arm 34 (as shown), which acquires 2D x-rayimages. For some applications, such 2D images are acquired by an imagingdevice (such as an o-arm or a 3D x-ray c-arm) situated in the operatingroom and also capable of generating 3D images. For example, such imagingdevice may be used for generating 3D image data at the beginning of theintervention in order to image the baseline anatomy in 3D, and thenagain at the latter part of the intervention in order to evaluate itsoutcomes (such as how well implants were positioned), and in between beused similarly to a regular c-arm in order to generate 2D during theintervention. For some applications, such device fulfils both the rolesof the 3D CT and the 2D c-arm, as such roles are described throughoutthis document with respect to embodiments of the present invention.

For some applications, the 2D images are captured in real time by aframe grabber of system 20 that is connected to an output port of theC-arm. Alternatively or additionally, system 20 and the C-arm areconnected to one another via a PACS network (or other networkingarrangement, wired or wireless) to which system 20 and C-arm 34 areconnected, and the 2D images are transferred, once acquired, to system20 via the PACS network (e.g., via memory 24). Alternatively oradditionally, the C-arm sends image files, for example in DICOM format,directly to system 20 (e.g., via memory 24).

Typically, the interventional part of a procedure that is performed onskeletal anatomy, such as the spine, commences with the insertion of atool, such as a Jamshidi™ needle 36 which is typical forminimally-invasive (or less-invasive) surgery. A Jamshidi™ needletypically includes an inner tube and an outer tube. The Jamshidi™ needleis typically inserted to or towards a target location, at which pointother tools and/or implants are inserted using the Jamshidi™ needle.Typically, in open surgery, for lower-diameter tools and/or implants,the inner tube of the Jamshidi™ needle is removed, and the tool and/orimplant is inserted via the outer tube of the Jamshidi™ needle, whilefor larger-diameter tools and/or implants, the tool and/or implant isinserted by removing the inner tube of the Jamshidi™ needle, inserting astiff wire through the outer tube, removing the outer tube, and theninserting the tool and/or implant along the stiff wire. Forminimally-invasive surgery, the aforementioned steps (or similar stepsthereto) are typically performed via small incisions. Alternatively, formore-invasive or open surgery, the tool inserted may be, for example, apedicle finder and/or a pedicle marker.

It is noted that, in general throughout the specification and the claimsof the present application, the term “tool” should be interpreted asincluding any tool or implant that is inserted into any portion of theskeletal anatomy during a procedure that is performed upon the skeletalanatomy. Such tools may include flexible, rigid and/or semi-rigidprobes, and may include diagnostic probes, therapeutic probes, and/orimaging probes. For example, the tools may include Jamshidi™ needles,other needles, k-wires, pedicle finders, pedicle markers, screws, nails,other implants, implant delivery probes, drills, endoscopes, probesinserted through an endoscope, tissue ablation probes, laser probes,balloon probes, injection needles, tissue removal probes, drug deliveryprobes, stimulation probes, dilators, etc. Typically, such proceduresinclude spinal stabilization procedures, such as vertebroplasty (i.e.,injection of synthetic or biological cement in order to stabilize spinalfractures), kyphoplasty (i.e., injection of synthetic or biologicalcement in order to stabilize spinal fractures, with an additional stepof inflating a balloon within the area of the fracture prior toinjecting the cement), fixation (e.g., anchoring two or more vertebraeto each other by inserting devices such as screws into each of thevertebrae and connecting the screws with rods), fixation and fusion(i.e., fixation with the additional step of an implant such as a cageplaced in between the bodies of the vertebrae), biopsy of suspectedtumors, tissue ablation (for example, RF or cryo), injection of drugs,and/or endoscopy (i.e., inserting an endoscope toward a vertebra and/ora disc, for example, in order to remove tissue (e.g., disc tissue, orvertebral bone) that compresses nerves).

Reference is now made to FIG. 2, which is a schematic illustration oftwo Jamshidi™ needles 36 being inserted into a vertebra 38, as used inprior art techniques. Typically, a spinal intervention aimed at avertebral body is performed with tools being aimed at 10-11 o'clock and1-2 o'clock insertion windows with respect to the subject's spine. Toolinsertion into a vertebra should avoid the spinal cord 42, andadditionally needs to avoid exiting the vertebra from the sides, leavingonly two narrow insertion windows 44, on either side of the vertebra. Asdescribed hereinbelow with reference to FIGS. 3A-4B, typically the mostimportant images for determining the locations of the insertion windowsare those derived from 3D image data, and are not available from thereal time 2D images that are typically acquired during the intervention.

Reference is now made to FIGS. 3A and 3B, which are schematicillustrations of a 3D CT image of a vertebra (FIG. 3A), as well as a 2Daxial slice that is derived from the 3D CT image (FIG. 3B), such imagesbeing used in prior art techniques. Reference is also made to FIGS. 4Aand 4B, which show C-arm 34 being used to acquire an anterior-posterior(“AP”) 2D radiographic image and a resultant AP image (FIG. 4A), andC-arm 34 being used to acquire a lateral 2D radiographic image and aresultant lateral image (FIG. 4B), as used in prior art techniques.

As may be observed, the view of the vertebra that is important fordetermining the entry point, insertion direction, and insertion depth ofthe tool is shown in the axial 2D image slice of FIG. 3B. By contrast,the 2D radiographic images that are acquired by the C-arm are summationsof 3D space, and do not show cross-sectional views of the vertebra.Furthermore, due to the anatomy of the human body, such summations wouldnot have been valuable if and when acquired from an axial angle (fromthe head, or from the toes) because it would not be possible to discernspecifically the spinal portion being operated upon. As describedhereinabove, Computer Aided Surgery (CAS) systems typically make use ofCT and/or MRI images, generated before the subject has been placed inthe operating room, or once the subject has been placed in the operatingroom but typically before an intervention has commenced. However, suchprocedures are typically more expensive than non-CAS procedures (suchnon-CAS procedures, including open procedures, mini-open procedures, andminimally-invasive procedures), limit tool selection to those fittedwith location sensors as described above, and typically require suchtools to be individually identified and calibrated at the beginning ofeach surgery.

In accordance with some applications of the present invention, theintra-procedural location of a tool is determined with respect to 3Dimage data (e.g., a 3D image, a 2D cross-section derived from 3D imagedata, and/or a 2D projection image derived from 3D image data), in anon-CAS procedure (e.g., in an open, mini-open and/or minimally-invasiveprocedure). The techniques described herein are typically practicedwithout requiring the fitting of location sensors (such as infraredtransmitters or reflectors, or magnetic or electromagnetic sensors) tothe tool or to the subject, and without requiring identification and/orcalibration of tools prior to the procedure. The techniques describedherein are typically practiced without requiring the fitting of anyradiopaque marker to the tool, rather they rely on the existingradio-opacity of the tool for its identification in the x-ray images.The techniques described herein are typically practiced withoutrequiring knowledge of the precise geometry and/or the dimensions of thetool for its identification in the x-ray images. The techniquesdescribed herein typically do not require tracking the location of thesubject's body or the applicable portion of the subject's body, and donot assume any knowledge of the location coordinates of the subject'sbody in some reference frame. The techniques described herein typicallydo not require location sensors that rely upon tracking technologies(e.g., electromagnetic or IR tracking technologies) that are nottypically used in an orthopedic operating room when not using CASsystems. Further typically, the techniques described herein arepracticed without requiring knowledge of any precise parameters of anyindividual pose of the 2D radiographic imaging device (e.g., C-arm 34),and typically without requiring poses of the 2D radiographic imagingdevice (e.g., C-arm 34) to be tracked relative to each other, and/orrelative to the position of the subject. For some applications, 2Dradiographic images (e.g., 2D x-ray images) are acquired from two ormore views, by moving a radiographic imaging device to respective posesbetween acquisitions of the images of respective views. Typically, asingle x-ray source is used for acquisition of the 2D x-ray images,although, for some applications, multiple sources are used. In general,where views of the 2D radiographic imaging device are described hereinas being AP, lateral, oblique, etc., this should not be interpreted asmeaning that images must be acquired from precisely such views, ratheracquiring images from generally such views is typically sufficient.Typically, the techniques described herein are tool-neutral, i.e., thetechniques may be practiced with any applicable tool and typicallywithout any modification and/or addition to the tool.

It is noted that although some applications of the present invention aredescribed with reference to 3D CT imaging, the scope of the presentinvention includes using any 3D imaging, e.g., MRI, 3D x-ray imaging, 3Dultrasound imaging, and/or other modalities of 3D imaging, mutatismutandis. Such imaging may be performed prior to, at the commencementof, and/or at some point during, an intervention. For example, the 3Dimaging may be performed before the subject has been placed within theoperating room, when the subject is first placed within the operatingroom, or at some point when the subject is in the operating room, butprior to the insertion of a given tool into a given target portion, etc.Similarly, although some applications of the present invention aredescribed with reference to 2D radiographic or x-ray imaging, the scopeof the present invention includes using any 2D imaging, e.g., ultrasoundand/or other modalities of 2D imaging, mutatis mutandis. Although someapplications of the present invention are described with reference toprocedures that are performed on skeletal anatomy and/or vertebrae ofthe spine, the scope of the present invention includes applying theapparatus and methods described herein to other orthopedic interventions(e.g., a joint (e.g., shoulder, knee, hip, and/or ankle) replacement,joint repair, fracture repair (e.g., femur, tibia, and/or fibula), aprocedure that is performed on a rib (e.g., rib removal, or ribresection), vascular interventions, cardiovascular interventions,neurovascular interventions, abdominal interventions, diagnosticinterventions, therapeutic irradiations, and/or interventions performedon other portions of a subject, including interventions in which 3Dimage data are acquired prior to the intervention and 2D images areacquired during the intervention, mutatis mutandis.

Reference is now made to FIGS. 5A-D, which are schematic illustration ofsets 50 of radiopaque markers 52 which are typically placed in thevicinity of a skeletal portion, e.g., spine, of a subject, either incontact with or not in contact with the body of the subject, inaccordance with some applications of the present invention. For someapplications, sets 50 of radiopaque markers 52 includes a support 53having a series of discretely identifiable support-affixed radiopaquemarkers 52, such that the series of discretely identifiablesupport-affixed radiopaque markers 52 appear in radiographic images ofthe skeletal portion. For some applications, the series of markers 52 isa series of sequential discretely identifiable support-affixedradiopaque markers along a longitudinal axis of support 53. For someapplications, the sets of markers are disposed on a drape disposed onthe applicable portion of the subject's body, for example, an incisiondrape attached upon the applicable portion, as shown. The drape istypically sterile and disposable. For some applications, the set ofmarkers includes an authentication and/or an anti-counterfeitingelement, such as RFID, bar code(s), etc.

Typically, sets 50 of markers 52 are attached, e.g., by an adhesivedisposed on a surface of the marker, e.g., an adhesive disposed onsupport 53, to a surface of the subject in a vicinity of a site, e.g.,skeletal portion, at which an intervention is to be performed, and suchthat at least some of the markers appear in 2D radiographic images thatare acquired of the intervention site from typical imaging views forsuch an intervention. For example, for a procedure that is performed onthe subject's vertebra(e) and particularly within one or more vertebralbodies, the markers are typically placed on the subject's back in avicinity of the site of the spinal intervention, such that at least someof the markers appear in 2D radiographic images that are acquired of theintervention site from AP imaging views, and potentially from additionalimaging views as well. For some applications, the markers are placed onthe subject's side in a vicinity of the site of the spinal intervention,such that at least some of the markers appear in 2D radiographic imagesthat are acquired of the intervention site from a lateral imaging view.For some applications, the markers are placed on the subject's back,such that at least some of the markers are level with the subject'ssacrum.

For some applications, known dimensions of, or distances between (e.g.,markers spaced at 1 cm from other another), radiopaque markers 52 areused in scaling 2D x-ray images comprising portions of the marker setprior to the registration of such 2D images with a 3D data set. Suchregistration is further described hereinbelow. Typically, and as knownin the art, scaling of the images to be registered, when performed priorto the actual registration, facilitates the registration.

For some applications, the set of markers comprises an arrangementwherein portions thereof are visible from different image views. Forsome applications, such arrangement facilitates for the surgeon theintra-procedural association of elements, including anatomical elementssuch as a vertebra, seen in a first x-ray image acquired from one view,for example AP, with the same elements as seen in a second x-ray imageacquired from a second view, for example lateral. For some applications,such association is performed manually by the surgeon referring to theradiopaque markers and identifying markers that have a known associationwith one another in the x-ray images, e.g., via matching of alphanumericcharacters or distinct shapes. Alternatively or additionally, theassociation is performed automatically by computer processor 22 ofsystem 20 by means of image processing.

Using known techniques, such association between images, for example ofa particular vertebra seen on those images, often requires inserting atool into or near to, or placing a tool upon, a vertebra of interestsuch that the tool identifies that vertebra in both images.

According to embodiments of the present invention, association betweenimages acquired from different views (for example AP and lateral, or APand oblique, or lateral and oblique) is facilitated by any of thefollowing techniques:

-   -   For some applications, a marker set 50 comprise 3D radiopaque        elements of different identifiable shapes that may be identified        from multiple views. In such case, a same 3D element is        typically identifiable from multiple viewing angles.        Consequently, a same vertebra situated at, near or relative to        such 3D elements may be identified in images acquired from        different viewing angles.    -   For some applications, a radiopaque marker set 50 comprises at        least one 2D object, e.g., segment (for example, label element,        or foldable tabs 54 having at least one tab-affixed radiopaque        marker 52′) that when unfolded is visible from a first image        view (e.g., most views except for lateral, e.g., AP), and when        folded away from the body of the subject, e.g., upwards, is        visible from both the first image view and a second image view        that is different from the first image view by at least 10        degrees, e.g., at least 20 degrees, e.g., at least 30 degrees        (e.g., lateral). Typically, the 2D foldable segments, e.g.,        tabs, have no adhesive disposed on them. For some applications a        fold line of tab(s) 54 is parallel to the longitudinal axis of        the support. For some applications, the surgeon may fold upwards        at any given moment only those one or more foldable 2D segments,        e.g., tabs 54 that he or she wishes to be visible from the        lateral direction. For example, the surgeon may fold the 2D        foldable segments subsequently to acquiring the radiographic        image from the first view and prior to acquiring the        radiographic image from the second view. Alternatively, the 2D        foldable segment may be folded prior to the start of the        procedure. Typically, such foldable arrangement also facilitates        manufacturing the markers by printing radiopaque ink on support        53, e.g., a flat surface or sheet.    -   For some applications, such as is shown in FIG. 5B, radiopaque        marker set 50 comprises elements that may be converted (for        example by folding) from 2D (for example a flat printed marker)        to 3D such that in the 3D form an element is identifiable        concurrently from multiple angles. For example, the at least one        2D foldable segment, e.g., tab 54 may be converted to a 3D        element 54′ when folded away from the surface of the subject        such that 3D element 54′ appears in radiographic images acquired        from at least the first and second image views, e.g., from both        AP and lateral image views. For some applications, the at least        one 2D foldable segment, e.g., tab 54 is shaped to define at        least one slit, e.g., at least two slits, that facilitates the        2D foldable segment converting to 3D element 54′.

For some applications, radiopaque marker set 50 is in the form of aframe-like label, such as is shown in FIG. 5C, in which certain 2Delements, e.g., segments or tabs 54, when unfolded, are visible from atop view, and when folded are visible from both a top view and a sideview, in accordance with some applications of the present invention.Typically, such arrangement also facilitates the manufacturing of themarker which can be done by printing the radiopaque ink on a flatsurface.

Reference is now made to FIG. 5E. Typically, as depicted by the flowchart in FIG. 5E, at least one radiopaque marker 52, e.g., set 50 ofmarkers 52, is attached to a surface of the subject in the vicinity ofthe skeletal portion, e.g., spine (step 212), a radiographic image isacquired from a first image view of the skeletal portion and theradiopaque marker 52 (step 214), and a radiographic image is acquiredfrom a second image view (step 216) of the skeletal portion and the atleast one 2D foldable segment, e.g., tab 54, when the 2D foldablesegment is folded away from the surface of the subject. Typically, thesecond image view is different from the first image view by at least 10degrees, e.g., at least 20 degrees, e.g., at least 30 degrees. Afteracquiring the first and second radiographic images, a given skeletalportion, e.g., a given vertebra of a spine, that appears in theradiographic images of the skeletal portion from the first image viewmay be associated with the given skeletal portion in the radiographicimages of the skeletal portion from the second image view, byidentifying the at least one folded 2D segment in the radiographicimages acquired from the first and second image views.

Typically, surgery on skeletal anatomy commences with attaching asterile surgical drape, typically an incision drape, at and around thesurgical site. In the case of spinal surgery, the surgical approach maybe anterior, posterior, lateral, oblique, etc., with the surgical drapeplaced accordingly. For such applications, sets 50 of markers 52 aretypically placed above the surgical drape. Alternatively, sets ofmarkers are placed on the subject's skin (e.g., if no surgical drape isused). For some applications, sets of markers are placed under thesubject's body, on (e.g., attached to) the surgical table, and/or suchthat some of the markers are above the surgical table in the vicinity ofthe subject's body. For some applications, a plurality of sets ofmarkers are used. For example, multiple sets of markers may be placedadjacently to one another. Alternatively or additionally, one or moresets of markers may be placed on the subject's body such that at leastsome markers are visible in each of a plurality of x-ray image views,e.g., on the back or stomach and/or chest for the AP or PA views, and onthe side of the body for the lateral view. For some applications, asingle drape with markers disposed thereon extends, for example, fromthe back to the side, such that markers are visible in both AP andlateral x-ray image views.

For some applications, a first marker set 50 a and second marker set 50b are placed on the subject's body such that, at each (or most) imagingview applied during the procedure for the acquisition of images, atleast one of the first and second markers (or a portion thereof) isvisible in the acquired images. For example, such as is shown in FIG.5D, in the case of spinal procedures with a dorsal approach, first andsecond marker sets 50 a and 50 b may be placed at the left and rightsides of the patient's spine, respectively, and directionally along thespine.

For some applications, only a first set of markers is placed on thesubject's body, typically at a position (e.g., along the spine) thatenables it to be visible from each (or most) imaging view applied duringthe procedure for the acquisition of images.

For some applications, a first marker set 50 a and a second marker set50 b are each modular. For example, a marker in the form of a notchedruler, may comprise several ruler-like modules. Typically, the number ofmodules to be actually applied to the subject's body is related to theoverall size of the subject, to the location of the targeted vertebra(e)relative to the anatomical reference point (e.g., sacrum) at whichplacement of the marker sets begins, or to a combination thereof. Forexample, a target vertebra in the lumbar spine may require one module, atarget vertebra in the lower thoracic spine may require two modules, atarget vertebra in the upper thoracic spine may require three modules,etc.

Typically, the sets of markers are positioned on either side of thesubject's spine such that even in oblique x-ray image views of theintervention site (and neighboring portions of the spine), at leastradiopaque markers belonging to one of the sets of markers are visible.Further typically, the sets of markers are positioned on either side ofthe subject's spine such that even in zoomed-in views acquired from thedirection of the tool insertion, or in views that are oblique (i.e.,diagonal) relative to the direction of tool insertion, at leastradiopaque markers belonging to one of the sets of markers are visible.Typically, the sets of radiopaque markers are placed on the subject,such that the radiopaque markers do not get in the way of either AP orlateral x-ray images of vertebrae, such that the radiopaque markers donot interfere with the view of the surgeon during the procedure, and donot interfere with registration of 2D and 3D image data with respect toone another (which, as described hereinbelow, is typically based ongeometry of the vertebrae).

For some applications, the sets of markers as shown in FIG. 5C are usedin open-surgery procedures where a large central incision is made alongthe applicable portion of the spine. For such procedures, a relativelylarge central window is required for performing the procedure betweenthe two sets of markers. For some applications, the sets of markers asshown in FIG. 5C are used in less invasive, or minimally invasive,surgery as well.

Radiopaque markers 52 are typically in the form of markings (e.g.,lines, notches, numbers, characters, shapes) that are visible to thenaked eye (i.e., the markings are able to be seen without specialequipment) as well as to the imaging that is applied. Typically, themarkers are radiopaque such that the markers are visible in radiographicimages. Further typically, markers that are placed at respectivelocations with respect to the subject are identifiable. For example, asshown in FIGS. 5A and 5B respective radiopaque alphanumeric charactersare disposed at respective locations. For some applications, markersplaced at respective locations are identifiable based upon otherfeatures, e.g., based upon the dispositions of the markers relative toother markers. Using a radiographic imaging device (e.g., C-arm 34), aplurality of radiographic images of the set of radiopaque markers areacquired, respective images being of respective locations along at leasta portion of the subject's spine and each of the images including atleast some of the radiopaque markers.

For some applications, all markings in the marker set are visible bothin the x-ray images (by virtue of being radiopaque) and to the naked eye(or optical camera). For some applications, some elements of the markerset are not radiopaque, such that they are invisible in the x-ray imagesand yet visible to the naked eye (or camera). For example, a centralruler placed on the subject's body may have notches or markings thatcorrespond directly to those of one or both sets of markers that are tothe side(s), and yet unlike the latter sets of markers it is notradiopaque. For some applications, when the marker set is placeddorsally, such a ruler facilitates for the surgeon the localization ofspecific spinal elements (e.g., vertebrae) when looking at the subject'sback and yet does not interfere with the view of those same spinalelements in the x-ray images.

The marker set may include a series of discretely identifiable, e.g.,distinct, radiopaque symbols (or discernible arrangements ofradio-opaque markers), such as is shown in FIG. 5A. For someapplications the series of markers may be a series of sequentialdiscretely identifiable radiopaque markers. For some applications, suchsymbols assist in the stitching of the individual x-ray images into thecombined images, by providing additional identifiable registrationfiducials for matching a portion of one image with portion of anotherimage in the act of stitching the two images together.

For some applications, sets 50 of markers 52, and/or a rigid radiopaquejig are used to facilitate any one of the following functionalities:

-   -   Vertebra level verification, as described hereinbelow.    -   Arriving at a desired vertebra intra-procedurally, without        requiring needles to be stuck into the patient, and/or counting        along a series of non-combined x-rays.    -   Displaying a 3D image of the spine that includes indications of        vertebra thereon, using vertebral level verification.    -   Determining the correct incision site(s) prior to actual        incision(s).    -   Identifying changes in a pose of the 2D imaging device (e.g.,        the x-ray C-arm) and/or a position of the patient. Typically, if        the position of the 2D imaging device relative to the subject,        or the position of the subject relative to the 2D imaging        device, has changed, then in the 2D images there would be a        visible change in the appearance of the markers 52 relative to        the anatomy within the image. For some applications, in response        to detecting such a change, the computer processor generates an        alert. Alternatively or additionally, the computer processor may        calculate the change in position, and account for the change in        position, e.g., in the application of algorithms described        herein. Further alternatively or additionally, the computer        processor assists the surgeon in returning the 2D imaging device        to a previous position relative to the subject. For example, the        computer processor may generate directions regarding where to        move an x-ray C-arm, in order to replicate a prior imaging        position, or the computer processor may facilitate visual        comparison by an operator.    -   Providing a reference for providing general orientation to the        surgeon throughout a procedure.    -   Providing information to the computer processor regarding the        orientation of image acquisition and/or tool insertion, e.g.,        anterior-posterior (“AP”) or posterior-anterior (“PA”), left        lateral or right lateral, etc.    -   Generating and updating a visual roadmap of the subject's spine,        as described in further detail hereinbelow.

For some applications, at least some of the functionalities listed aboveas being facilitated by use of sets 50 of markers 52, and/or a rigid jigare performed by computer processor 22 even in the absence of sets 50 ofmarkers 52, and/or a rigid jig, e.g., using techniques as describedherein. Typically, sets 50 of markers 52, and/or a rigid jig are usedfor level verification, the determination of a tool entry point or anincision site, performing measurements using rigid markers as areference, identifying changes in a relative pose of the 2D imagingdevice (e.g., the x-ray C-arm) and of the subject, and providing generalorientation. All other functionalities of system 20 (such asregistration of 2D images to 3D image data and other functionalitiesthat are derived therefrom) typically do not necessarily require the useof sets 50 of markers 52, and/or a rigid jig. The above-describedfunctionalities may be performed automatically by computer processor 22,and/or manually.

Applications of the present invention are typically applied, in non-CAS(the term “non-CAS” also refers to not in the current form of CAS at thetime of the present invention) spinal surgery, to one or more proceduraltasks including, without limitation:

-   -   Applying pre-operative 3D visibility (e.g., from CT and/or MRI),        or 3D visibility gained via image acquisition within the        operating room, during the intervention. It is noted that 3D        visibility provides desired cross-sectional images (as described        in further detail hereinbelow), and is typically more        informative and/or of better quality than that provided by        intraoperative 2D images. (It is noted that, for some        applications, intraoperative 3D imaging is performed.)    -   Confirming the vertebra(e) to be operated upon.    -   Determining the point(s) of insertion of one or more tools.    -   Determining the direction of insertion of one or more tools.    -   Monitoring tool progression, typically relative to patient        anatomy, during insertion.    -   Reaching target(s) or target area(s).    -   Exchanging tools while repeating any of the above steps.    -   Determining tool/implant position within the anatomy, including        in 3D.    -   Generating and updating a visual roadmap of the subject's spine,        as described in further detail hereinbelow.

Reference is now made to FIG. 6, which is a flowchart showing steps thatare typically performed using system 20, in accordance with someapplications of the present invention. It is noted that some of thesteps shown in FIG. 6 are optional, and some of the steps may beperformed in a different order to that shown in FIG. 6. In a first step70, targeted vertebra(e) are marked by an operator with respect to 3Dimage data (e.g., a 3D image, a 2D cross-section derived from 3D imagedata, and/or a 2D projection image derived from 3D image data) of thesubject's spine. For some applications, in a second step 72, sets 50 ofmarkers 52 are placed on the subject, underneath the subject, on thesurgical table, or above the surgical table in a vicinity of thesubject. For some applications, step 72 is performed prior to step 70.Typically, in a third step 74, vertebrae of the spine are identified inorder to verify that the procedure is being performed with respect tothe correct vertebra (a step which is known as “level verification”),using radiographic images of the spine and the markers to facilitate theidentification, or alternatively, using registration of 2D x-ray imageswith the 3D image data, as further described hereinbelow. In a fourthstep 76, an incision site, e.g., a skin-level incision site, (typicallyin the case of minimally-invasive or less-invasive surgery) or a toolentry point, e.g., a skeletal-portion-level entry point, (typically inthe case of open surgery) is determined. Throughout this document, theterm “incision site” (or “site of incision”) refers to the site ofmaking an incision of limited size, typically in the course ofminimally-invasive and less-invasive surgery, while the term “entrypoint” (or “point of entry”) typically refers to a point at which a toolenters a targeted skeletal element such as a vertebra. However, the twoterms may be also used interchangeably when describing certainapplications of the present invention, or for example an incision sitemay be referred to as a skin-level insertion point. For someapplications, in a fifth step 78, the first tool in the sequence oftools (which in less-invasive surgery is often a needle, e.g., aJamshidi™ needle) is typically inserted into the subject (e.g., in thesubject's back), and is slightly fixated in the vertebra.

For some applications, in step 78 a tool (which in more-invasive surgeryis often a pedicle finder) is not yet inserted but rather is positionedrelative to a vertebra, wherein such vertebra is often partially exposedat such phase, either manually or using a holder device that istypically fixed to the surgical table. Such holder device typicallyensures that the subsequent acquisition in step 80 of two or more 2Dradiographic images prior to actual tool insertion are with the tool ata same position relative to the vertebra. For some applications, motionof the applicable portion of the subject in between the acquisition ofthe two or more images is detected by means of a motion detection sensoras described later in this document. For some applications, if motion isdetected that the acquisition of pre-motion images may be repeated.

In a sixth step 80, two or more 2D radiographic images are acquired fromrespective views that typically differ by at least 10 degrees, e.g., atleast 20 degrees (and further typically by 30 degrees or more), and oneof which is typically from the direction of insertion of the tool.Typically, generally-AP and generally-lateral images are acquired.Alternatively or additionally, images from different views are acquired.In a seventh step 82, computer processor 22 of system 20 typicallyregisters the 3D image data to the 2D images, as further describedhereinbelow.

Subsequent to the registration of the 3D image data to the 2D imagesadditional features of system 20 as described in detail hereinbelow maybe applied by computer processor 22. For example, in step 84, thecomputer processor drives display 30 to display a cross-section derivedfrom the 3D image data at a current location of the tip of a tool asidentified from a 2D image, and, optionally, to show a vertical line onthe cross-sectional image indicating a line within the cross-sectionalimage somewhere along which the tip of the tool is currently disposed.

It is noted, that, as described in further detail hereinbelow, for someapplications, in order to perform step 84, the acquisition of one ormore 2D x-ray images of a tool at a first location inside the vertebrais from only a single x-ray image view, and the one or more 2D x-rayimages are registered to the 3D image data by generating a plurality of2D projections from the 3D image data, and identifying a 2D projectionthat matches the 2D x-ray images of the vertebra. In response toregistering the one or more 2D x-ray images acquired from the singlex-ray image view to the 3D image data, the computer processor drives adisplay to display a cross-section derived from the 3D image data at athe first location of a tip of the tool, as identified from the one ormore 2D x-ray images, and optionally to show a vertical line on thecross-sectional image indicating a line within the cross-sectional imagesomewhere along which the first location of the tip of the tool isdisposed. Typically, when the tip of the tool is disposed at anadditional location with respect to the vertebra, further 2D x-rayimages of the tool at the additional location are acquired from the samesingle x-ray image view, or a different single x-ray image view, and theabove-described steps are repeated. Typically, for each location of thetip of the tool to which the above-described technique is applied, 2Dx-ray images need only be acquired from a single x-ray image view, whichmay stay the same for the respective locations of the tip of the tool,or may differ for respective locations of the tip of the tool.Typically, two or more 2D x-rays are acquired from respective views, andthe 3D image data and 2D x-ray images are typically registered to the 3Dimage data (and to each other) by identifying a corresponding number of2D projections of the 3D image data that match respective 2D x-rayimages. In step 86, the computer processor drives display 30 to displaythe anticipated (i.e., extrapolated) path of the tool with reference toa target location and/or with reference to a desired insertion vector.In step 88, the computer processor simulates tool progress within asecondary 2D imaging view, based upon observed progress of the tool in aprimary 2D imaging view. In step 90, the computer processor overlays animage of the tool, a representation thereof, and/or a representation ofthe tool path upon the 3D image data (e.g., a 3D image, a 2Dcross-section derived from 3D image data, and/or a 2D projection imagederived from 3D image data), the location of the tool or tool pathhaving been derived from current 2D images.

Reference is now made to FIG. 7, which shows a vertebra 91 designatedupon a coronal cross-sectional image 92 and upon a sagittalcross-sectional image 94 of a subject's spine, the cross-sectionalimages being derived from 3D image data, in accordance with someapplications of the present invention. For some applications, suchimages are the Preview images that are often generated at the beginningof a 3D scan. For some applications, such images are derived from the 3Dscan data, such as by using a DICOM Viewer. As described hereinabovewith reference to step 70 of FIG. 6, typically prior to the subjectbeing placed into the operating room, or while the subject is in theoperating room but before an intervention has commenced, an operatormarks the targeted vertebra(e) with respect to the 3D image data (e.g.,a 3D image, a 2D cross-section derived from 3D image data, and/or a 2Dprojection image derived from 3D image data). For some applications, inresponse to the operator marking one vertebra, the computer processordesignates additional vertebra(e). For some applications, the operatormarks any one of, or any combination of, the following with respect tothe 3D image data: a specific target within the vertebra (such as afracture, a tumor, etc.), desired approach directions/vectors for toolinsertion as will be further elaborated below, and/or desired placementlocations of implants (such as pedicle screws). For some applications,the operator marks the targeted vertebra with respect to a 2D x-rayimage that has a sufficiently large field of view to encompass anidentifiable portion of the anatomy (e.g., the sacrum) and the targetedvertebra(e). For some applications, more than one targeted vertebra ismarked, for example vertebrae that are to be fixated to and/or fused toone another, and for some applications, two or more vertebra(e) that arenot adjacent to one another are marked.

For some applications, the computer processor automatically counts thenumber of vertebrae on the image from an identifiable anatomicalreference (e.g., the sacrum) to the marked target vertebra(e). It isthen known that the targeted vertebra(e) is vertebra N from theidentifiable anatomical reference (even if the anatomical labels of thevertebra(e) are not known). For some applications, the vertebra(e) arecounted automatically using image-processing techniques. For example,the image-processing techniques may include shape recognition ofanatomical features (of vertebrae as a whole, of traverse processes,and/or of spinous processes, etc.). Or, the image-processing techniquesmay include outer edge line detection of spine (in a 2D image of thespine) and then counting the number of bulges along the spine (eachbulge corresponding to a vertebra). For some applications, theimage-processing techniques include techniques described in US2010-0161022 to Tolkowsky, which is incorporated herein by reference.For some applications, the vertebra(e) are counted manually by theoperator, starting with the vertebra nearest the anatomical referenceand till the targeted vertebra(e).

Referring to step 72 of FIG. 6 in more detail, for some applications, inwhich a procedure is performed on a given vertebra of the subject'sspine, one or more sets 50 of radiopaque markers 52 are placed upon ornear the subject, such that markers that are placed at respectivelocations with respect to the subject are identifiable, e.g., as shownin FIGS. 5A-C. For example, as shown in FIGS. 5A and 5B respectiveradiopaque alphanumeric characters are disposed at respective locations.For some applications, markers placed at respective locations areidentifiable based upon other features, e.g., based upon thedispositions of the markers relative to other markers. Using aradiographic imaging device (e.g., C-arm 34), a plurality ofradiographic images of the set of radiopaque markers are acquired,respective images being of respective locations along at least a portionof the subject's spine and each of the images including at least some ofthe radiopaque markers. Using computer processor 22, locations of theradiopaque markers within the radiographic images are identified, bymeans of image processing. At least some of the radiographic images arecombined with respect to one another based upon the identified locationsof the radiopaque markers within the radiographic images. Typically,such combination of images is similar to stitching of images. However,the images are typically not precisely stitched such as to stitchportions of the subject's anatomy in adjacent images to one another.Rather, the images are combined with sufficient accuracy to be able todetermine a location of the given vertebra N within the combinedradiographic images.

For some applications, based upon the combined radiographic images, thecomputer processor automatically determines a location of the givenvertebra (e.g., the previously-marked targeted vertebra) within thecombined radiographic images. For some applications, the computerprocessor automatically determines location of the given vertebra withinthe combined radiographic images by counting the number of vertebrae onsaid image from an identifiable anatomical reference (e.g., the sacrum).For some applications, the counting is performed until theaforementioned N. For some applications, the counting is performed untila value that is defined relative to the aforementioned N. For someapplications, the vertebra(e) are counted automatically usingimage-processing techniques. For example, the image-processingtechniques may include shape recognition of anatomical features (ofvertebrae as a whole, of traverse processes, and/or of spinousprocesses, etc.). Or, the image-processing techniques may include outeredge line detection of spine (in a 2D image of the spine) and thencounting the number of bulges along the spine (each bulge correspondingto a vertebra). For some applications, the image-processing techniquesinclude techniques described in US 2010-0161022 to Tolkowsky, which isincorporated herein by reference. For some applications, the computerprocessor facilitates manual determination of the location of the givenvertebra within the combined radiographic images by displaying thecombined radiographic images. For some applications, based upon thecombined radiographic images, the operator manually determines,typically by way of counting vertebrae upon the combined images startingat the anatomical reference, a location of the given vertebra (e.g., thepreviously-marked targeted vertebra) within the combined radiographicimages.

For some applications, the marker sets as observed in the stitched x-rayimages are overlaid, typically automatically and by means of imageprocessing, upon the corresponding CT images of the spine or of theapplicable spinal portions. For some applications, that facilitatessubsequent matching by the user between corresponding skeletal elementsin the stitched x-ray and in the CT images.

Reference is now made to FIG. 8A, which is a flow chart showing theabovementioned method for level verification with the additional step218 of positioning an intraoperative 3D imaging device. Based upon thelocation of the given vertebra within the combined radiographic images,a location of the given vertebra in relation to the set of radiopaquemarkers that is placed on or near the subject is determined. Anintraoperative 3D imaging device can then be positioned such that animaging volume of the 3D imaging device at least partially overlaps thegiven vertebra.

It is noted that in the absence of sets 50 of markers 52, the typicalmethodology for determining the location of a given vertebra includesacquiring a series of x-rays along the patient's spine from the sacrum,and sticking radiopaque needles into the subject in order to match thex-rays to one another. Typically, in each x-ray spinal image only 3-4vertebrae are within the field of view, and multiple, overlapping imagesmust be acquired, such as to enable human counting of vertebra using theoverlapping images. This technique may also involve switching back andforth between AP and lateral x-ray images. This method is oftentime-consuming and radiation-intensive.

A known clinical error is wrong-level surgery, as described, forexample, in “Wrong-Site Spine Surgery: An Underreported Problem? AAOSNow,” American Association of Orthopedic Surgeons, March 2010. Thatfurther increases the desire for facilitating level verification byapplications of the present invention, as described herein.

Reference is now made to FIG. 8B which shows an example of a 3D CT image95 of a subject's spine displayed alongside a combined radiographicimage 96 of the subject's spine, in accordance with some applications ofthe present invention. In accordance with some applications of thepresent invention, set 50 of markers 52 is placed upon or near thesubject, such that the bottom of the set of markers is disposed over, orin the vicinity of, the sacrum. (and in particular the upper portionthereof, identifiable in the x-ray images). A sequence of x-ray imagesfrom generally the same view as one another are acquired along thespine, typically, but not necessarily, with some overlap betweenadjacent images. Typically, the specific pose of the x-ray C-arm whenacquiring each of the images is not known, the C-arm is not tracked by atracker nor are its exact coordinates relative to the subject's body(and more specifically the applicable portion thereof) known. Thesequence of x-ray images is typically acquired from a generally-AP view,but may also be acquired from a different view, such as agenerally-lateral view. Using computer processor 22, locations of theradiopaque markers within the radiographic images are identified, bymeans of image processing. At least some of the radiographic images arecombined with respect to one another based upon the identified locationsof the radiopaque markers within the radiographic images. For example,combined radiographic image 96 is generated by combining (a) a firstx-ray image 97 acquired from a generally-AP view and which starts at thesubject's sacrum and which includes markers H-E of the right marker setand markers 8-5 of the left marker set with (b) second x-ray image 98acquired from a generally similar view to the first view (but one whichis not exactly the same) and which includes markers E-B of the rightmarker set and markers 5-2 of the left marker set. As noted previously,the aforementioned markers may be alphanumeric, or symbolic, or both.

(It is noted that in FIG. 8B the alphanumeric markers appear as white inthe image. In general, the markers may appear as generally white orgenerally black, depending on (a) the contrast settings of the image(e.g., do radiopaque portions appear as white on a black background, orvice versa), and (b) whether the markers are themselves radiopaque, orthe markers constitute cut-outs from a radiopaque backing material, asis the case, in accordance with some applications of the presentinvention.)

Typically, the combination of images is similar to stitching of images.However, the images are often not precisely stitched such as to stitchportions of the subject's anatomy in adjacent images to one another.Rather, the images are combined with sufficient accuracy to facilitatecounting vertebrae along the spine within the combined image. Thephysical location of a given vertebra is then known by virtue of itbeing adjacent to, or in the vicinity of, or observable in the x-rayimages relative to, a given one of the identifiable markers. It is notedthat in order to combine the radiographic images to one another, thereis typically no need to acquire each of the images from an exact view(e.g., an exact AP or an exact lateral view), or for there to be exactreplication of a given reference point among consecutive images. Rather,generally maintaining a given imaging direction, and having at leastsome of the markers generally visible in the images is typicallysufficient.

As described hereinabove, for some applications, the computer processorautomatically counts (and, for some applications, labels, e.g.,anatomically labels, and/or numerically labels) vertebrae within thecombined radiographic images in order to determine the location of thepreviously-marked target vertebra(e), or other vertebra(e) relative tothe previously marked vertebra. Alternatively, the computer processordrives the display to display the combined radiographic images such asto facilitate determination of the location of the previously-markedtarget vertebra(e) by an operator. The operator is able to count to thevertebra within the combined radiographic images, to determine, withinthe combined images, which of the radiopaque markers are adjacent to orin the vicinity of the vertebra, and to then physically locate thevertebra within the subject by locating the corresponding physicalmarkers.

Reference is now made to FIG. 8C, which shows an example of a 3D CTimage 100 of the subject's spine displayed alongside a 2D radiographicimage 102 of the subject's spine, in accordance with some applicationsof the present invention. As shown, markers 52 appear on the combinedradiographic image. As shown, vertebra 91, which was identified by anoperator with respect to the 3D image data (as described hereinabovewith reference to FIG. 8A), has been identified within the 2Dradiographic image using the above-described techniques, and is denotedby cursor 104.

For some applications, a spinal CT image data (in 3D or a 2D slice)matching the viewing direction from which the x-ray images were acquiredis displayed concurrently with the stitched x-ray images. For example,in the case of x-ray images acquired from a generally-AP direction, acoronal CT view is displayed. For some applications, the x-ray images,or the stitched x-ray image, are interconnected with the CT image suchthat when the user (or the system) selects a vertebra on the x-ray, thesame vertebra is indicated/highlighted on the CT image, or vice versa.For some applications, such connection is generated by registering oneor more DRRs of the spine as a whole, or of the corresponding spinalsection, or of one or more individual vertebrae, with the x-ray imagesor stitched image. For some applications, such connection is generatedby other means of image processing, including in accordance withtechniques described hereinabove in the context of counting vertebrae.

For some applications, generation of the combined image includesblending the edges of individual x-ray images from which the combinedimage is generated, typically resulting in a more continuous-lookingcombined image.

Reference is now made to FIGS. 8D-I, which show, in accordance withapplications of the present invention, an example of generating acombined spinal image from four individual x-ray images, shownrespectively in FIGS. 8D-G, that were acquired sequentially along thespine, with each individual x-ray image showing a portion of the spine.Marker sets 50, each comprising a numbered radio-opaque ruler coupledwith identifiable arrangements of radio-opaque elements, are placedalong both sides of the spine. A portion of one or both marker sets 50is visible in each of the x-ray images in FIGS. 8D-G. The combined imageshown in FIG. 8H was generated by stitching the four x-ray images withthe help of marker sets 50 and in accordance with techniques describedhereinabove. For generating the combined image shown in FIG. 8I,blending was applied to corresponding edges of x-ray images of FIGS.8D-G.

For some applications, 2D x-ray images of the subject's spine, or of aportion thereof, are stitched into a combined image, or are relatedspatially to one another without actually stitching them, by using 3Dimage data of the subject's spine (or of a portion thereof) as a“bridge,” and as described hereinbelow.

For some applications, the 3D image data comprises all of the spinalportions visible in the x-ray images. For some applications, the 3Dimage data comprises only some of the spinal portions visible in thex-ray images.

For some applications, a plurality of 2D x-ray images are acquired,respective images being of respective locations along at least a portionof the subject's spine. For some applications, all images are acquiredfrom a similar viewing angle, for example an angle that is approximatelyAP. For some applications, images are acquired from different viewingangles.

For some applications, some or all of the images are acquired with someoverlap between consecutive two images with respect to the skeletalportion visible in each of them. For some applications, some or all ofthe images are acquired with small gaps (typically a portion of avertebra) between consecutive two images with respect to the skeletalportion visible in each of them.

For some applications, the images are stitched to one another, typicallywithout using radiopaque markers, and while using the subject's 3D imagedata, to provide a combined image of the spine or of a portion thereof,by a computer processor that performs the following:

-   -   i. Each newly-acquired x-ray image is registered with 3D image        data of the subject's spine, using Digitally Reconstructed        Radiographs (DRRs) as described by embodiments of the present        invention.    -   ii. As a result, vertebrae visible in each x-ray image become        associated with corresponding vertebrae in the 3D image data.    -   iii. As a result, for example: vertebrae that are visible, in        whole or in part, in both x-ray images, are identified as being        the same vertebrae; alternatively or additionally, vertebrae        that are visible in any two images relating to neighboring        portions of the spine, are identified with respect to their        anatomical positions relative to one another.    -   iv. The two x-ray images are now stitched such that vertebrae        (or portions of vertebrae) visible in each of the images are now        overlaid upon one another, or the images are placed along one        another in a manner that represents the subject's anatomy, all        in accordance with the positions of those vertebrae along the        subject's spine.

Alternatively of additionally, the vertebrae visible in each of thex-ray images are marked as such upon the 3D image data. For someapplications, the vertebrae visible in each x-ray image may be relatedto, or marked on, a sagittal view, or a sagittal cross-section, of the3D image data. For some applications, the vertebrae visible in eachx-ray image may be marked on a coronal view, or a coronal cross-section,of the 3D image data.

For example, if vertebrae L5, L4, L3 and L2 are visible in a first x-rayimage, and vertebra L2, L1, T12 and T11 are visible in a second x-rayimage:

-   -   The first x-ray image and the second x-ray image, typically if        acquired from similar views, are stitched to one another with        vertebra L2 (or a portion thereof) being the overlapping        section, typically creating a combined image;    -   Alternatively, the first x-ray image and the second x-ray image,        typically if acquired from non-similar views, are displayed        relative to one another such that vertebra L2 (or a portion        thereof) is at a parallel position in both;    -   Alternatively, the first x-ray image and the second x-ray image        are displayed as related to a sagittal view, or a sagittal        cross-section, of the 3D image data for vertebra L5 through T11,        such that the first x-ray image is related, typically visually,        to vertebrae L5 through L2 and the second x-ray image is        related, typically visually, to vertebrae L2 through T11;    -   Alternatively, the first x-ray image and the second x-ray image        are displayed as related to a coronal view, or a coronal        cross-section, of the 3D image data for vertebra L5 through T11,        such that the first x-ray image is related, typically visually,        to vertebrae L5 through L2 and the second x-ray image is        related, typically visually, to vertebrae L2 through T11.

Alternatively, for example, if vertebrae L5, L4, L3 and L2 are visiblein a first x-ray image, and vertebra L1, T12, T11 and T10 are visible ina second x-ray image:

-   -   The first x-ray image and the second x-ray image are displayed        relative to one another such that vertebra L2 in the first x-ray        image is adjacent to vertebra L1 in the second x-ray image;    -   The first x-ray image and the second x-ray image are displayed        within a combined image, relative to one another such that        vertebra L2 in the first x-ray image is adjacent to vertebra L1        in the second x-ray image within the combined image;    -   Alternatively, the first x-ray image and the second x-ray image        are displayed as related to a sagittal view, or a sagittal        cross-section, of the 3D image data for vertebra L5 through T10,        such that the first x-ray image is related, typically visually,        to vertebrae L5 through L2, and the second x-ray image is        related, typically visually, to vertebrae L1 through T10;    -   Alternatively, the first x-ray image and the second x-ray image        are displayed as related to a coronal view, or a coronal        cross-section, of the 3D image data for vertebra L5 through T11,        such that the first x-ray image is related, typically visually,        to vertebrae L5 through L2 and the second x-ray image is        related, typically visually, to vertebrae L1 through T10.

For some applications, the techniques described hereinabove are furtherapplied for level verification, optionally in combination with othertechniques described herein.

Thus, reference is now made to FIG. 32, which is a flow chart showing amethod for image stitching, in accordance with some applications of thepresent invention, and comprising the following steps:

(i) acquiring 3D image data of a skeletal portion (step 356),

(ii) acquiring a plurality of 2D radiographic images, each image showinga distinct segment of the skeletal portion (step 358)

(iii) registering the 2D radiographic images with the 3D image data,such that a post-registration correspondence is created between each 2Dradiographic image and the 3D image data (step 360),

(iv) using the post-registration correspondence between each of the 2Dradiographic images and the 3D image data, relating the 2D images withrespect to each other (step 360), and

(v) using the relationship of the 2D radiographic images with respect toeach other, generating a combined 2D radiographic image comprisingmultiple segments of the skeletal portion (step 362).

Reference is now made to FIGS. 9A-B, which are flow charts showinganother method for performing “level verification,” in accordance withsome applications of the present invention. After acquiring 3D imagedata of at least a target vertebra (step 220), using at least onecomputer processor, the targeted vertebra is indicated within the 3Dimage data (step 222). A radiopaque element that is typically alsovisible to the naked eye, e.g., the tip of a surgical tool, such as ascalpel, or set 50 of radiopaque markers 52, is positioned on the bodyof the subject (step 224) with respect to the spine such that theradiopaque element appears in 2D radiographic images that acquired ofthe spine (step 226). Computer processor 22 then identifies the targetedvertebra in the 2D radiographic image by registering the targetedvertebra in the 3D image data to the targeted vertebra in the 2Dradiographic image (step 228). As shown by the flowchart in FIG. 9B, todo the registration, computer processor 22 attempts to register eachvertebra that is visible in the 2D radiographic image to the targetedvertebra in the 3D image data until a match is found by generating aplurality of 2D projections of the targeted vertebra from the 3D imagedata (step 230) and for each vertebra that is visible in the 2Dradiographic image, identifying if there exists a 2D projection of thetargeted vertebra that matches the 2D radiographic image of thatvertebra (step 232). Once the targeted vertebra has been identified itis indicated on the 2D radiographic image (step 234) such that alocation of the targeted vertebra is now identified with respect toradiopaque element.

It should also be noted that level verification using embodiments of thepresent invention is also useful for correctly positioning a 3D imagingdevice (such as an O-arm or a 3D x-ray device), situated within theoperating room, relative to the subject's body and prior to an actual 3Dscan. A common pre-operative CT or MRI device is, according to thespecific scan protocol being used, typically configured to scan along anentire body portion such as a torso. For example, such scan may includethe entire lumbar spine, or the entire thoracic spine, or both. Incontrast, the aforementioned 3D imaging devices available inside someoperating rooms, at the time of the present invention, have a verylimited scan area, typically a cubical volume whose edges are each 15-20cm long. Thus, correct positioning of such 3D imaging device prior tothe scan relative to the subject's spine, and in particular relative tothe targeted spinal elements, is critical for ensuring that the targetedvertebra(e) are indeed scanned. For some applications, levelverification using aforementioned embodiments of the present inventionyields an indication to the operator of those visible elements of themarker set, next to which the 3D imaging device should be positioned forscanning the spinal segment desired to be subsequently operated upon,such that an imaging volume of the 3D imaging device at least partiallyoverlaps the targeted vertebra. For some applications, in the operatingroom, the targeted vertebra(e) are level-verified using embodiments ofthe present invention and then the 3D imaging device is positioned suchthat its imaging volume (whose center is often indicated by a red lightprojected upon the subject's body, or some similar indication) coincideswith the targeted vertebra(e). For example, if the marker set is anotched ruler placed on the subject's body along the spine, then usingembodiments of the present invention the operator may realize that the3D imaging device should be positioned such that its red light isprojected on the subject's body at a level that is in between notches #7and #8 of the ruler.

For some applications, when a vertebra is selected in an x-ray image(acquired at any phase of the medical procedure) or a combined x-rayimage, a 3D image of the same vertebra is displayed automatically. Forsome applications, the 3D vertebral image auto-rotates on the display.For some applications, the 3D vertebral image is displayed with somelevel of transparency, allowing the user to observe tools inserted inthe vertebra, prior planning drawn on the vertebra, etc. the selectionof the vertebra may be by the user or by the system. The autorotationpath (i.e., the path along which the vertebra rotates) may be 2D or 3D,and may be system-defined or user-defined. The level of transparency maybe system-defined or user-defined. The same applies not only tovertebrae, but also to other spinal or skeletal elements.

For some applications, based upon counting and/or labeling of thevertebrae in the combined radiographic image, computer processor 22 ofsystem 20 counts and/or labels vertebrae within the 3D image data (e.g.,a 3D image, a 2D cross-section derived from 3D image data, and/or a 2Dprojection image derived from 3D image data). For some applications, thecomputer processor drives the display to display the labeled vertebraewhile respective corresponding 2D images are being acquired anddisplayed. Alternatively or additionally, the computer processor drivesthe display to display the labeled vertebrae when the combinedradiographic image has finished being generated and/or displayed. It isnoted that, typically, the computer processor counts, labels, and/oridentifies vertebrae on the 3D image data and on the 2D radiographicimages without needing to determine relative scales of the 3D image dataand 2D images. Rather, it is sufficient for the computer processor to beable to identify individual vertebrae at a level that is sufficient toperform the counting, labeling, and/or identification of vertebrae.

It is noted that the above-described identification of vertebrae that isfacilitated by markers 52 is not limited to being performed by thecomputer processor at the start of an intervention. Rather, the computerprocessor may perform similar steps at subsequent stages of theprocedure. Typically, it is not necessary for the computer processor torepeat the whole series of steps at the subsequent stages, since thecomputer processor utilizes knowledge of an already-identified vertebra,in order to identify additional vertebrae. For example, afteridentifying and then performing a procedure with respect to a firstvertebra, the computer processor may utilize the combined radiographicimage to derive a location of a further target vertebra (which may beseparated from the first vertebra by a gap), based upon thealready-identified first vertebra. For some applications, in order toderive the location of a further target vertebra, the computer processorfirst extends the combined radiographic image (typically, using themarkers in order to do so, in accordance with the techniques describedhereinabove).

Reference is now made to FIG. 10, which shows an example of an opticalimage 110 displayed alongside a 2D radiographic (e.g., x-ray) image 112,in accordance with some applications of the present invention. Asdescribed with reference to step 76 of FIG. 6, subsequent to identifyinga target vertebra along the subject's spine, typically, the operatordetermines a desired site for an incision or tool insertion. For someapplications, in order to facilitate the determination of the incisionsite or tool insertion site, an optical camera 114 is disposed withinthe operating room such that the optical camera has a generally similarviewing angle to that of the 2D radiographic imaging device. Forexample, the camera may be disposed on x-ray C-arm 34, as shown inFIG. 1. Alternatively or additionally, the camera may be disposed on aseparate arm, may be handheld, may be the back camera of a display suchas a tablet or mini-tablet device, and/or may be held by another memberof the operating room staff. For some applications, the camera is placedon the surgeon's head. Typically, for such applications, the surgeonuses a head-mounted display.

For some applications, a 2D radiographic image 112 of a portion of thesubject's body is acquired in a radiographic imaging modality, using the2D radiographic imaging device (e.g., C-arm 34), and an optical image110 of the subject's body is acquired in optical imaging modality, usingoptical camera 114 (shown in FIG. 1). Computer processor 22 of system 20identifies radiopaque markers (e.g., markers 52) in the radiographicimage and in the optical image, by means of image processing. By way ofexample, in FIG. 10, radiopaque gridlines and alphanumeric radiopaquemarkers associated with the radiopaque gridlines are visible in both theradiographic and the optical image. Based upon the identification of theradiopaque markers in the radiographic image and in the optical image,the computer processor bidirectionally maps the radiographic image andthe optical image with respect to one another. It is noted thatacquisition of the radiographic image and the optical image fromgenerally-similar views (but not necessarily identical views) istypically sufficient to facilitate the bidirectional mapping of theimages to one another, by virtue of the radiopaque markers that arevisible in both of the images.

For some applications, the radiographic image and the optical image arefused with one another and displayed as a joint image. For someapplications, any of the images is adjusted (e.g. scaled, distorted,etc.), typically according to elements of the marker set observed inboth images, prior to such fusion. For some applications, only the x-rayimage is displayed to the operator, with the location of the tool (e.g.,knife) positioned upon the subject identified from the optical image andmarked upon the x-ray image.

As shown in FIG. 10, for some applications, the computer processordrives display 30 to display the radiographic image and the opticalimage separately from one another, upon one or more displays.Subsequently, in response to receiving an input indicating a location ina first one of the radiographic and the optical images, the computerprocessor generates an output indicating the location in the other oneof the radiographic and the optical images. For example, in response toa line or a point being marked on 2D x-ray image 112, the computerprocessor indicates a corresponding lines or points overlaid on theoptical image 110. Similarly, in response to a line or a point beingmarked on optical image 110, the computer processor indicates acorresponding lines or points overlaid on the 2D x-ray image 112.Further similarly, in response to a line or a point being marked on, oran object such as a k-wire or incision knife laid upon, the subject'sbody (e.g., back in the case of a planned dorsal tool insertion) as seenin a then-current optical image 110, the computer processor identifiessuch line, point or object (or applicable portion thereof) and indicatesa corresponding lines or points overlaid on the 2D x-ray image 112. Forsome applications, a line or point is drawn on the subject's body (e.g.,on the subject's back in the case of a planned dorsal tool insertion)using radiopaque ink.

Traditionally, in order to determine the location of an incision site, arigid radiopaque wire (such as a K-wire) is placed on the subject's backat a series of locations, and the x-rays are taken of the wire at thelocations, until the incision site is determined. Subsequently, a knifeis placed at the determined incision site, and a final x-ray image isacquired for verification. By contrast, in accordance with the techniquedescribed herein, initially a single x-ray image may be acquired andbidirectionally mapped to the optical image. Subsequently the wire isplaced at a location, and the corresponding location of the wire withrespect to the x-ray image can be observed (using the bidirectionalmapping) without requiring the acquisition of a new x-ray image.Similarly, when an incision knife is placed at a location, thecorresponding location of an applicable portion of the knife (typically,its distal tip) with respect to the x-ray image can be observed (usingthe bidirectional mapping) without requiring the acquisition of a newx-ray image. Alternatively or additionally, a line can be drawn on thex-ray image (e.g., a vertical line that passes along the vertebralcenters, anatomically along the spinous processes of the vertebrae) andthe corresponding line can be observed in the optical image overlaid onthe patient's back.

It should be noted however that for some applications, and in theabsence of an optical camera image of the subject, the marker set thatis visible both in the x-ray images and upon the subject's body servesas a joint reference for when identifying insertion points or incisionsites by the surgeon. Typically, such identification is superior withrespect to time, radiation, iterations, errors, etc., compared withcurrent practices (such as in common non-CAS surgical settings) prior tothe present invention.

For some applications, a surgeon places a radiopaque knife 116 (oranother radiopaque tool or object) at a prospective incision site(and/or places a tool at a prospective tool insertion location) andverifies the location of the incision site (and/or tool insertionlocation) by observing the location of the tip of the knife (or portionof another tool) with respect to the x-ray (e.g., via cursor 117), bymeans of the bi-directional mapping between the optical image and thex-ray image. For some applications, the functionalities describedhereinabove with reference to FIG. 10, and/or with reference otherfigures, are performed using markers (which are typically sterile),other than markers 52. For example, a radiopaque shaft 118, ruler,radiopaque notches, and/or radiopaque ink may be used.

Reference is now made to FIG. 11, which shows an example of a 2Dradiographic (e.g., x-ray) image 120 displayed alongside across-sectional image 122 of a subject's vertebra that is derived from a3D image data of the vertebra, in accordance with some applications ofthe present invention. For some applications, even prior to registeringthe 2D images to the 3D image data (as described hereinbelow), thefollowing steps are performed. X-ray image 120 of a given view thesubject's spine (e.g., AP, as shown) is acquired. A point is indicatedupon the image, e.g., the point indicated by cursor 124 in FIG. 11.Computer processor 22 of system 20 automatically identifies the endplates of the vertebra and calculates the relative distance of indicatedpoint from end plates. (It is noted that the computer processortypically does not calculate absolute distances in order to perform thisfunction.) From the 3D (e.g., CT) image of the same vertebra, thecomputer processor generates and displays a cross-section of a givenplane (which is typically axial) at the indicated location (e.g. image122). For some applications, upon the cross-section, the computerprocessor drives the display to show a line 126 (e.g., a vertical line)within the cross-section, the line indicating that the indicatedlocation falls somewhere along the line. For some applications, the lineis drawn vertically upon an axial cross-section of the vertebra asshown. The computer processor determines where to place the lineaccording to distance of the indicated point from left and right edgesof the vertebra, and/or according to the position of the indicated pointrelative to visible features (e.g., spinous process, traverse processes,pedicles) in the x-ray image. Typically, the cross-sectional image withthe line, and coupled with the surgeon's tactile feel of how far fromthe vertebra the skin is (and/or deriving such information from a 3Dimage), assists the surgeon in calculating the desired insertion angleof a tool.

Referring again to step 78 of FIG. 6, the first tool in the sequence oftools (which is typically a needle, e.g., a Jamshidi™ needle, for lessinvasive surgery, or a pedicle finder for more open surgery) is insertedinto the subject (e.g., in the subject's back), and is slightly fixatedin the vertebra. Subsequently, in step 80 of FIG. 6, two or more 2Dradiographic images are acquired from respective views that typicallydiffer by at least 10 degrees, e.g., at least 20 degrees, e.g., 30degrees, and one of which is typically from the direction of insertionof the tool. Common combinations of such views include AP and left orright lateral, AP with left or right oblique, left oblique with leftlateral, and right oblique with right lateral. It is noted that for someapplications, 2D radiographic images of the tool and the vertebra areacquired from only a single x-ray image view.

Reference is now made to FIGS. 12A-J which are schematic illustrationsand a flowchart of a method for determining a designated, e.g., planned,point 235 for skin-level or skeletal-portion-level incision/entry. Forsome applications, step 70 of FIG. 6 comprises not only marking targetedvertebra(e), but also planning the paths for tool insertion, includingdetermining the intended site(s) of entering the patient's body with thetool, typically at skin-level or at a skeletal-portion-level, e.g.,spine-level.

For some applications, the determination of intended incision/entrysite, i.e., designated point 235, includes the following steps for eachtargeted vertebra, with each step either performed manually by theoperator or automatically. (It is noted that some of the steps areoptional, and that some of the steps may be performed in a differentorder to that listed below.)

-   -   1. For each targeted vertebra, 3D scan data of the vertebra is        acquired and loaded (step 236 in FIG. 12B).    -   2. Scan data is displayed and viewed, typically at the coronal,        sagittal and axial planes (such as is shown in FIG. 13A).        Typically, the viewer software automatically ties (which can        also be thought of as “links” or “associates”) the three views        to one another, such that manipulating the viewing in one plane        effects corresponding changes in the views in the other planes.        Optionally, a 3D reconstructed view is added.    -   3. The viewing planes are adjusted such that the vertebra is        typically viewed in the axial view from a direction that is        axial relative to the specific vertebra (as opposed to being        axial to the longitudinal axis of the spine as a whole, since        each vertebra may have its own angle relative to the        longitudinal axis of the spine as a whole).    -   4. Vertebral axial cross-sections are leafed through.    -   5. An axial cross-section 238 most suitable for tool insertion        is selected. In other words, an axial cross-section that would        typically be the cross-section on which, during actual tool        insertion, the longitudinal centerline of the tool would ideally        reside, and thus where currently the planned approach vector        would reside. For the planned insertion of pedicle screws, that        would typically be an axial cross-section where the pedicles are        relatively large and thus suitable for screw insertion, and        further typically the largest for that vertebra. (In some cases,        that may be a different cross section for each of the two        pedicles of a same vertebra of the subject.) For some        applications, the insertion plane for the specific vertebra, or        pedicle within the vertebra, is selected in the sagittal view        and then the axial view is auto-aligned with that direction.    -   6. Pedicle length and width are measured for later section of        the specific tool or implant that will be used.    -   7. A generally-vertical line 240 is drawn upon such axial        cross-section, through the spinous process and all the way to        the skin. (Appropriate window-level values, such that the skin        is visible, are typically used when viewing the image data.)    -   8. Diagonal tool-insertion lines 242 are drawn upon axial        cross-section 238 through the pedicle, and typically both        pedicles of the vertebra, from inside the vertebral body to skin        level and potentially further beyond outside the subject's body.        Intersection points 244 of such lines 242 with the skin are        identified, i.e., at least one skin-level incision point or        skeletal-portion-level entry point is designated within the body        of the subject (step 250 in FIG. 12B). For some applications,        intersection points 244 are identified at both sides of the        vertebra. Alternatively, for some applications, only one        diagonal tool-insertion line 242 is drawn upon axial        cross-section 238, corresponding to one side of the vertebra,        and one intersection point 244 of line 242 with the skin is        identified.    -   9. (As noted previously, the two lines may reside on different        planes and thus different cross-sections.) Typically, each line        242 begins at skin level and ends at the designated target        within the vertebral body. Typically, each line 242 includes a        skin-level starting point, and entry point into the pedicle, an        exit point from the pedicle, or any combination thereof    -   10. Horizontal distances D1 and D2 of each of the intersection        points to the vertical line marking the spinous process are        measured and noted on the image.    -   11. Insertion angles (coronal, axial) for each tool-insertion        line, at the skin-level intersection point, are measured and        noted on the image.    -   12. Tool (and/or implant) representations are placed along one        or more insertion lines in order to select optimal tool sizes        (for example, the lengths and diameters of pedicle screws to be        inserted).    -   13. The aforementioned intersection points, e.g., skin-level        points (and potentially also the lines, angles and distances)        are associated and stored with the 3D scan data for that        vertebra (step 252 in FIG. 12B). The skin-level entry points or        incision sites are typically stored as 3D coordinates within        such 3D scan data.

For some applications and pursuant to the above, in step 76 of FIG. 6the aforementioned 3D scan data for the vertebra, including theadditional planning information and in particular the designatedpoint(s) 235, i.e., skin-level incision site(s) orskeletal-portion-level, e.g., spin-level entry point(s), is registeredwith an x-ray image 246, typically from an AP view, that includes thesame vertebra, using techniques such as DRRs that are further describedin subsequent sections of this document. As a result, the designatedpoint(s) 235, i.e., skin-level entry points or incision sites, are nowdisplayed upon x-ray image 246. For some applications, using embodimentsof the present invention as described above for step 76, one or morepoints 235′ are subsequently auto-marked on a camera image 248 of thesubject's back and displayed to the operator.

For some applications, such as is shown in FIG. 12C, a distance D3 of anincision site, e.g., designated point 235, from one or more(typically-nearest) elements, e.g., markers 52, of the marker set 50 ismeasured, manually or automatically, and is measured and displayed onthe x-ray image to facilitate physical determination of the incisionsite (for example, the incision site may be 6 cm horizontally to theright from marker number 21 on the left ruler-like marker set). For someapplications, such as is shown in FIG. 12D, distance D3 is displayed notnumerically but by markings 254 (e.g., notches) that are overlaid on thex-ray image and are spaced at known intervals D4, for example 1 cm, fromone another.

For some applications, a camera image is not available, and the operatorestimates, or measures physically, the locations of points 235′ on thesubject's back relative to the marker set that is (a) placed on thesubject's back and (b) also visible in the x-ray image. For someapplications, based on the location of the designated point with respectto the radiopaque element on the 2D radiographic image, the operatorlabels a location of the designated point on the subject's body.

For some applications, such as is shown in FIG. 12E, visualidentification of previously-planned skin-level incision sites, as wellas the desired direction of insertion towards each correspondingvertebral entry point upon the subject, may be performed with nomeasurements from the aforementioned planning provided upon the x-rayimage. In step 256 of FIG. 12B the operator places a radiopaque element258, such as the tip of a radiopaque tool, such as an incision knife, atthe estimated location of point 235′ on the subject's back (usingpoint(s) 235 in x-ray image 246 as a guide). In step 260 of FIG. 12B,the operator acquires an intraoperative x-ray image 262 (FIG. 12F),typically the same AP as before. In step 264 the intraoperative x-rayimage 262 is registered to the 3D image data such that the prior 3Dplanning data is auto-registered to x-ray image 262 (FIG. 12G) and bothpoint(s) 235 and radiopaque element 258, e.g., the knife's tip, are thusdisplayed in second x-ray image 262 (step 266). The operator can nowtell whether the knife is placed correctly, or whether another iterationis required. FIGS. 12H-I show a second iteration after the operator hasmoved radiopaque element 258, e.g., the knife's tip, closer to point235.

For some applications, such as is shown in FIG. 12J, in the absence of acurrent optical camera image of the subject, any of points 235 isfurther indicated upon the registered x-ray image as an intersection oftwo virtual lines 268 drawn relative to corresponding portions of themarket set. For some applications, the lines are generatedautomatically. For some applications, the lines are drawn manually bythe operator. Using marker sets 50 as a reference, the operator can nowreplicate the two virtual lines by laying long objects 270 (e.g.,K-wires, rulers, etc.) on the subject and marking the point 272 ofintersection. Consequently, the intersection of the K-wires indicatesthe planned skin-level insertion point 235′ on the subject.

Reference is now made to FIGS. 12K-P, which depict a method foridentification of an incision site with respect to radiopaque element258, in accordance with some applications of the present invention. Forsome applications, a first x-ray image 366 is acquired after positioningradiopaque element 258, e.g., the tip of a radiopaque tool, at anestimated location on the subject's back (FIG. 12K). On the left side ofFIG. 12K, 3D image data for the target vertebra is displayed along withthe 3D planning data containing generally-vertical line 240, diagonaltool-insertion line(s) 242, and intersection point(s) 244 correspondingto designated point(s) 235 which correspond to the incision site(s).FIG. 12L shows a DRR 368, generated from the 3D image data, that matchesx-ray image 366. The same planning data as shown in FIG. 12K is shownagain side-by-side with DRR 368 in FIG. 12L. The planning data is thenassociated with, e.g., projected onto, DRR 368. For some applications,such as is shown in FIG. 12M, the planning data including intersectionpoint(s) 244 corresponding to designated point(s) 235 may be displayedon DRR 368. Alternatively, the association between the planning data andDRR 368 is maintained within computer processor 22.

FIG. 12N shows the planning data now projected onto x-ray image 366(x-ray image 366 matching DRR 368), such that the planning dataincluding intersection point(s) 244 corresponding to designated point(s)235 is now visible relative to radiopaque element 258, e.g., the tip ofthe radiopaque tool. If radiopaque element 258 is not in the correctposition, as is the case shown in FIG. 12N, radiopaque element 258 ismoved and a second x-ray image 370 acquired. FIG. 12O shows the secondx-ray image 370 side-by-side with the planning data. As shown in FIG.12P, the planning data is then projected onto second x-ray image 370using the same steps as described with reference to FIGS. 12L-N.Radiopaque element 258, such as the tip of the operating tool, can nowbe seen on target at designated point 235.

Reference is now made to FIGS. 13A-B, which show an example of planningtool insertion sites at skin level, as described hereinabove, upon the3D scan data, in accordance with embodiments of the present invention.FIG. 13A depicts generation and selection of an appropriate vertebralcross section 274 (FIG. 13B) that meets the aforementioned criteria,from the 3D scan data of a spine phantom with such data viewed in allthree planes as described above (axial view 276, coronal view 278, andsagittal view 280). For example, it should be noted that the line 282 inthe sagittal view indicates the vertebra is sliced axially in an axialdirection that is relative to the targeted vertebra itself (as opposedto being axial to the longitudinal axis of the spine as a whole). FIG.13B depicts generally vertical spinous-process line 240, two diagonalinsertion lines 242, and two skin-level insertion points 235, allgenerated in accordance with embodiments of the present invention asdescribed above with reference to FIG. 12A.

It should be noted that embodiments described hereinbelow are alsouseful for identifying the insertion point into a vertebra in the caseof more-invasive or open surgery, wherein the applicable portion of avertebra is visible via an incision, or exposed. For some applications,such determination of insertion points is performed according to thefollowing steps for each targeted vertebra, with each step performedmanually by the operator or automatically. (It is noted that some of thesteps are optional, and that some of the steps may be performed in adifferent order to that listed below.)

-   -   1. For each targeted vertebra, 3D scan data of the vertebra is        loaded.    -   2. Scan data is displayed and viewed, typically at the coronal,        sagittal and axial planes. Typically, the viewer software        automatically ties the three views to one another, such that        manipulating the viewing in one plane effects corresponding        changes in the views in the other planes. Optionally, a 3D        reconstructed view is added.    -   3. The viewing planes are adjusted such that the vertebra is        typically viewed in the axial view from an axial direction that        is relative to the specific vertebra (as opposed to being axial        to the longitudinal axis of the spine as a whole, since each        vertebra has its own typical angle relative to the longitudinal        axis of the spine as a whole).    -   4. Vertebral axial cross-sections are leafed through.    -   5. An axial cross-section most suitable for tool insertion is        selected. In other words, that would typically be the        cross-section on which, during actual tool insertion, the        longitudinal center line of the tool would ideally reside, and        where the currently planned approach vector would reside. For        the planned insertion of pedicle screws, that would typically be        an axial cross-section where the pedicles are relatively large        and thus suitable for screw insertion, and further typically the        largest for that vertebra. (In some cases, that may be a        different cross section for each of the two pedicles of the        vertebra within the subject.)    -   6. A generally-vertical line is drawn upon such axial        cross-section, through the spinous process and all the way to        the skin (Appropriate window-level values, such that the skin is        visible, are used.)    -   7. Diagonal tool-insertion lines are drawn upon such axial        cross-section through the pedicle, and typically both pedicles        of the vertebra, from inside the vertebral body to the        applicable boarder of the vertebra and potentially further        beyond outside the subject's body. Intersection points of such        lines with the skin, typically at both sides of the vertebra,        are identified.    -   8. Horizontal distances of each of the intersection points to        the vertical line marking the spinous process are measured and        noted on the image.    -   9. Insertion angles (coronal, axial) for each tool-insertion        line, at the vertebral-border intersection point, are measured        and noted on the image.    -   10. Tool representations are placed along one or more insertion        lines in order to select optimal tool sizes (for example, the        lengths and diameters of pedicle screws to be inserted).    -   11. The aforementioned entry points (and potentially also        angles, lines, distances) are associated and stored with the 3D        scan data for that vertebra. The skin-level entry points or        incision sites are typically stored as 3D coordinates within        such 3D scan data.    -   Such steps may be followed by any of the embodiments previously        described for skin-level insertion, by which the entry points        from the 3D data set are registered to the applicable x-ray        image, displayed upon that x-ray image, and used for determining        point(s) of entry into the vertebra during surgery.

For some applications, both the incision sites at the skin level, andthe entry points into the vertebra at the vertebra's applicable edge,are calculated in the 3D data, then registered to, and displayed upon,the 2D x-ray image, and then used for determining the skin-levelincision site and the direction of tool entry through that site,typically in accordance with techniques described hereinabove. For someapplications, the distance of the incision site from one or more(typically-nearest) elements of the marker set is measured manually orautomatically and displayed to facilitate physical determination of theincision site and/or entry point.

For some applications, planning in its various forms as describedhereinabove also comprises marking an out-of-pedicle point along theplanned insertion path. An out-of-pedicle point is at or near a locationalong the planned path where the object being inserted along the pathexits the pedicle and enters the vertebral body.

For some applications, one or more of the following points are markedalong the planned insertion path: incision at skin level, entry into thevertebra, out-of-pedicle, target, or any other point.

Reference is now made to FIGS. 14A and 14B, which show examples ofrespectively AP and lateral x-ray images of an elongated tool (such as aJamshidi™ needle) 36 being inserted into a subject's spine, inaccordance with some applications of the present invention. As shown,sets 50 of markers 52 typically appear at least in the AP image.

Reference is now made to FIGS. 15A and 15B, which show examples ofcorrespondence between views of a 3D image of a vertebra, with,respectively, first and second 2D x-ray images 132 and 136 of thevertebra, in accordance with some applications of the present invention.In FIG. 15A the correspondence between a first view 130 of a 3D image ofa vertebra with an AP x-ray image of the vertebra is shown, and in FIG.15B the correspondence between a second view 134 of a 3D image of avertebra with a lateral x-ray image of the vertebra is shown.

Reference is made to FIG. 16, which shows an example of a surgical tool284 gripped by an adjustable tool holder 286, with the tool holder fixedto the rail of a surgical table. For some applications, subsequent tothe fixation of the tool in the subject's vertebra, the 3D image dataand 2D images are registered to each other, in accordance with step 82of FIG. 6. For some applications, tool 284 is not fixated into thevertebra, but rather it is positioned relative to the vertebra. Anexample for a tool fixated in a vertebra is a needle inserted into avertebra in a less-invasive surgery performed through a small incision,while an example for a tool 284 positioned relative to a vertebra butnot inserted yet would be a pedicle finder aimed relative to thevertebra in the course of a more invasive surgery performed through alarge incision. For some applications, tool 284 is attached to toolholder 286 that can maintain tool 284 in a consistent position duringthe acquisition of one or more x-ray images. For some applications, suchholder 286 is attached to the surgical table, or to a separate stand, orto a robotic arm, or any combination thereof.

For some applications, holder 286 to which the tool is attached alsocomprises one or more angle gauges, typically digital. In such cases,the aforementioned insertion angles previously measured in the planningphase may be applied when aiming the tool at the vertebra. For someapplications, application of the angles is manual by the operator of theholder. For some applications, and when holder 286 is robotic,application of the angles is automated and mechanized. For someapplications, it is assumed that the applicable portion of the subjectis positioned completely horizontally.

However, it is noted that the registration of the 3D image data and the2D images to each other may be performed even in the absence of a toolwithin the images, in accordance with the techniques describedhereinbelow.

For some applications and when a tool is present in the 2D images butnot present in the 3D images, the visibility of a tool or a portionthereof is reduced (or eliminated altogether) by means of imageprocessing from the 2D images prior to their registration with the 3Dimage data. After registration is completed, 2D images with the toolpresent, i.e., as prior to the aforementioned reduction or elimination,are added to (utilizing the then-known registration parameters), orreplace, the post-reduction or elimination 2D images, within theregistered 2D-3D data, according to the registration already achievedwith the post-reduction or elimination 2D images. For some applications,regions in the 2D image comprising a tool or a marker set are excludedwhen registering the 2D images with the 3D data. For some applications,the aforementioned techniques facilitate registration of the 2D imageswith the 3D data set because all include at the time of theirregistration to one another only (or mostly) the subject's anatomy,which is typically the same, and thus their matches to one another neednot (or to a lesser extent) account for elements that are included inthe 2D images but are absent from the 3D data set. For someapplications, the reduction or elimination of the visibility of the toolor a portion thereof is performed using techniques and algorithmic stepsas described in US Patent Application 2015-0282889 to Cohen (andTolkowsky), which is incorporated herein by reference. The same appliesto a reduction of elimination of the visibility of previously-placedtools, such as implants (e.g., pedicle screws, rods, cages, etc.), inany of the images, such as prior to image registration.

Typically, the 3D image data and 2D images are registered to each otherby generating a plurality of 2D projections from the 3D image data andidentifying respective first and second 2D projections that match thefirst and second 2D x-ray images of the vertebra, as described infurther detail hereinbelow. (For some applications, 2D x-ray images frommore than two 2D x-ray image views are acquired and the 3D image dataand 2D x-ray images are registered to each other by identifying acorresponding number of 2D projections of the 3D image data that matchrespective 2D x-ray images.) Typically, the first and second 2D x-rayimages of the vertebra are acquired using an x-ray imaging device thatis unregistered with respect to the subject's body, by (a) acquiring afirst 2D x-ray image of the vertebra (and at least a portion of thetool) from a first view, while the x-ray imaging device is disposed at afirst pose with respect to the subject's body, (b) moving the x-rayimaging device to a second pose with respect to the subject's body, and(c) while the x-ray imaging device is at the second pose, acquiring asecond 2D x-ray image of at least the portion of the tool and theskeletal portion from a second view.

For some applications, the 3D imaging that is used is CT imaging, andthe following explanation of the registration of the 3D image data tothe 2D images will focus on CT images. However, the scope of the presentinvention includes applying the techniques describe herein to other 3Dimaging modalities, such as MRI and 3D x-ray, mutatis mutandis.

X-ray imaging and CT imaging both apply ionizing radiation to image anobject such as a body portion or organ. 2D x-ray imaging generates aprojection image of the imaged object, while a CT scan makes use ofcomputer-processed combinations of many x-ray images taken fromdifferent angles to produce cross-sectional images (virtual “slices”) ofthe scanned object, allowing the user to see inside the object withoutcutting. Digital geometry is used to generate a 3D image of the insideof the object from a large series of 2D images.

Reference is now made to FIGS. 17A, 17B, and 17C, which demonstrate therelationship between a 3D image of an object (which in the example shownin FIG. 17A is a cone) and side-to-side (FIG. 17B) and bottom-to-top(FIG. 17C) 2D images of the object, such relationship being utilized, inaccordance with some applications of the present invention. As shown,for the example of the cone, the bottom-to-top 2D image (which isanalogous to an AP x-ray image of an object acquired by C-arm 34, asschematically indicated in FIG. 17C) is a circle, while the side-to-sideimage (which is analogous to a lateral x-ray image of an object,acquired by C-arm 34, as schematically indicated in FIG. 17C) is atriangle. It follows that, in the example shown, if the circle and thetriangle can be registered in 3D space to the cone, then they alsobecome registered to one another in that 3D space. Therefore, for someapplications, 2D x-ray images of a vertebra from respective views areregistered to one another and to 3D image data of the vertebra bygenerating a plurality of 2D projections from the 3D image data, andidentifying respective first and second 2D projections that match the 2Dx-ray images of the vertebra.

In the case of 3D CT images, the derived 2D projections are known asDigitally Reconstructed Radiographs (DRRs). If one considers 3D CT dataand a 2D x-ray image of the same vertebra, then a simulated x-ray cameraposition (i.e., viewing angle and viewing distance) can be virtuallypositioned anywhere in space relative to a 3D image of the vertebra, andthe corresponding DRR that this simulated camera view would generate canbe determined. At a given simulated x-ray camera position relative tothe 3D image of the vertebra, the corresponding DRR that this simulatedcamera view would generate is the same as the 2D x-ray image. For thepurposes of the present application, such a DRR is said to match anx-ray image of the vertebra. Typically, 2D x-ray images of a vertebrafrom respective views are registered to one another and to 3D image dataof the vertebra by generating a plurality of DRRs from 3D CT image data,and identifying respective first and second DRRs (i.e., 2D projections)that match the 2D x-ray images of the vertebra. By identifyingrespective DRRs that match two or more x-ray images acquired fromrespective views, the x-ray images are registered to the 3D image data,and, in turn, the x-ray images are registered to one another via theirregistration to the 3D image data.

For some applications, and due to the summative nature of x-ray imaging,an x-ray image of a given vertebra may also, depending on the x-rayview, comprise elements from a neighboring vertebra. In such case, thoseelements may be accounted for (by way of elimination or inclusion)during the act of 2D-3D registration, and in accordance with embodimentsof the present invention. For some applications, such accounting for isfacilitated by 3D segmentation and reconstruction of the given(targeted) vertebra that is the focus of the then-current registrationprocess.

For some applications, x-ray images are enhanced using the correspondingDRRs from the 3D data set. For some applications, newly-acquired x-rayimages are enhanced by corresponding DRRs that were generated prior tothat in the act of registering previously-acquired x-ray images to thesame 3D data set. For some applications, the newly-acquired and thepreviously-acquired x-ray images are acquired from the same poses of thex-ray c-arm relative to the subject. For some applications, thenewly-acquired and the previously-acquired x-ray images are combinedwith one another for the purpose of image enhancement.

For some applications, in order to register the 2D images to the 3Dimage data, additional registration techniques are used in combinationwith the techniques described herein. For example, intensity basedmethods, feature based methods, similarity measures, transformations,spatial domains, frequency domains, etc., may be used to perform theregistration.

For some applications, and wherein the 3D image set was acquired in theoperating room, the 3D image set also comprises applicable portions ofmarker set(s) 50, such that the marker set serves as an additionalone-or-more registration fiducial in between the 2D images and the 3Ddata set.

Typically, by registering the x-ray images to the 3D image data usingthe above-described technique, the 3D image data and 2D x-ray images arebrought into a common reference frame to which they are all aligned andscaled. It is noted that the registration does not require tracking thesubject's body or a portion thereof (e.g., by fixing one or morelocation sensors, such as an IR light, an IR reflector, an opticalsensor, or a magnetic or electromagnetic sensor, to the body or bodyportion, and tracking the location sensors).

Typically, between preprocedural 3D imaging (e.g., 3D imaging performedprior to entering the operating room, or prior to performing a givenintervention) and intraprocedural 2D imaging, the position and/ororientation of a vertebra relative to the subject's body and toneighboring vertebrae is likely to change. For example, this may be dueto the patient lying on his/her back in preprocedural imaging but on thestomach or on the side for intraprocedural imaging, or the patient'sback being straight in preprocedural imaging, but being folded (e.g., ona Wilson frame) in intraprocedural imaging. In addition, in some cases,due to anesthesia the position of the spine changes (e.g. sinks), andonce tools are inserted into a vertebra, that may also change itspositioning relative to neighboring vertebrae. However, since a vertebrais a piece of bone, its shape typically does not change between thepreprocedural 3D imaging and the intraprocedural 2D imaging. Therefore,registration of the 3D image data to the 2D images is typicallyperformed with respect to individual vertebrae. For some applications,registration of the 3D image data to the 2D images is performed on aper-vertebra basis even in cases in which segmentation of a vertebra inthe 3D image data leaves some elements, such as portions of the spinousprocesses of neighboring vertebrae, within the segmented image of thevertebra. In addition, for some applications, registration of the 3Dimage data to the 2D images is performed with respect to a spinalsegment comprising several vertebrae. For example, registration of 3Dimage data to the 2D images may be performed with respect to a spinalsegment in cases in which the 3D image data were acquired when thesubject was already in the operating room and positioned upon thesurgical table for the intervention.

As described hereinabove, typically, during a planning stage, anoperator indicates a target vertebra within the 3D image data of thespine or a portion thereof (e.g., as described hereinabove withreference to FIG. 8A). For some applications, the computer processorautomatically identifies the target vertebra in the x-ray images, bymeans of image processing, e.g., using the techniques describedhereinabove. For some applications, the registration of the 3D imagedata to the 2D images is performed with respect to an individualvertebra that is automatically identified, by the computer processor, ascorresponding to a target vertebra as indicated by the operator withrespect to the 3D image data of the spine or a portion thereof (e.g., asdescribed hereinabove with reference to FIGS. 8B-C).

Typically, and since the registration is performed with respect to anindividual vertebra, the registration is not affected by motion of thevertebra that occurs between the acquisition of the two x-ray images(e.g., due to movement of the subject upon the surgical table, motiondue to respiration, etc.), since both motion of the C-arm and of thevertebra may be assumed to be rigid transformations (and thus, if bothmotions occur in between the acquisition of the two x-ray images, achaining of two rigid transformations may be assumed).

As described hereinabove, typically, 2D x-ray images of a vertebra fromrespective views are registered to one another and to a 3D image data ofthe vertebra by generating a plurality of DRRs from a 3D CT image, andidentifying respective first and second DRRs that match the 2D x-rayimages of the vertebra. By identifying respective DRRs that match two ormore x-ray images acquired from respective views, the x-ray images areregistered to the 3D image data, and, in turn, the x-ray images areregistered to one another via their registration to the 3D image data.

For some applications, in order to avoid double solutions when searchingfor a DRR that matches a given x-ray image, computer processor 22 firstdetermines whether the x-ray image is, for example, AP, PA, leftlateral, right lateral, left oblique, or right oblique, and/or fromwhich quadrant a tool is being inserted. The computer processor maydetermine this automatically, e.g., by means of sets 50 of markers 52,using techniques described herein. Alternatively, such information maybe manually inputted into the computer processor.

For some applications, in order to identify a DRR that matches a givenx-ray image, computer processor 22 first limits the search space withinwhich it is to search for a matching DRR, by applying the followingsteps. (It is noted that some of the steps are optional, and that someof the steps may be performed in a different order to that listedbelow.)

-   -   1. Information pertaining to the acquisition of the given x-ray        images is retrieved. Typically, such information includes the        angles of the different axes of the c-arm at the time of the        acquisition of the image. It should be noted that such angles        are typically relative to the base of the c-arm itself, not        relative to the subject's body and typically not even relative        to the surgical table (unless such table is integrated with the        c-arm, which is less common). Additionally, such information may        comprise the values of other imaging parameters (e.g., zoom        level) that may be of use for limiting the search space.        -   For some applications, the information is included in            standard (e.g., DICOM) image files generated by the x-ray            system, and such files are transferred from the x-ray system            to the processor, typically through a network connection.        -   For some applications, a capture device such as a frame            grabber, which is connected to the computer that comprises            processor 22, captures the screen image from the x-ray            system. Typically, such capture is upon or immediately after            the acquisition of the x-ray image and its display on the            native x-ray screen. Such screen image typically includes            not only the x-ray image but also additional (typically            textual) information such as the values of the            aforementioned different axes of the c-arm at the time of            the acquisition of the image. For some applications, such            values are read from the captured x-ray images by computer            processor 22 using Optical Character Recognition (OCR).        -   For some applications, computer processor 22 is fitted            previously with a configuration file pertaining to the model            of the x-ray system with such file including instructions on            the layout of the native x-ray screen including where each            textual data is located, and the use by the processor of            such file facilitates the identification of each desired            data item (such as the angular value of a specific axis of            the c-arm).        -   For some applications, such configuration file also includes            the values of other imaging parameters characterizing the            model of the x-ray system and/or the specific device, and is            not limited to information that appears on the native screen            of the x-ray system.    -   2. The angular values of the detectors of the CT scanner,        relative to the table on which the subject is positioned and        throughout the scan of the subject's body (or of the applicable        portion thereof), are typically included in the standard (e.g.,        DICOM) image files generated by the scanner and loaded onto the        computer that comprises processor 22.    -   3. For the generation of the DRRs from the CT data, the search        space is narrowed to a subset that is relatively close in its        viewing angles (typically relative to the scanner's table) to        the angles of the axes of the c-arm during the acquisition of        the x-ray image, and/or close with respect to other imaging        parameters.        -   For some applications, for example if the subject is            positioned on the back during the CT scan but on the stomach            at the time the x-ray image is acquired, proper translation            needs to be applied first, for example flipping the CT            angles up-down and/or left-right.

For some applications, in order to identify a DRR that matches a givenx-ray image, computer processor 22 first limits the search space withinwhich it is to search for a matching DRR, by identifying the marker setor elements thereof in the x-ray image and applying prior knowledge withwhich it was provided of what the marker set or its elements look likefrom different viewing directions, or at different zoom levels, or atdifferent camera openings, or any combination thereof. Typically, thesearch space is narrowed down to at or near simulated camerapositions/values from which the marker set or elements thereof are knownto appear in a similar manner to how they appear in the x-ray image.

For some applications, in order to identify a DRR that matches a givenx-ray image, some combination of techniques described in the presentapplication is applied.

For some applications, the registration of the 2D (e.g., x-ray) imageswith the 3D (e.g., CT) data is divided into a pre-processing phase andan online phase, i.e., during a medical procedure. Each of the twophases may be performed locally on a computer, or on a networkedcomputer, or via cloud computing, or by applying any combinationthereof.

Reference is now made to FIG. 30, which is a flow chart for a method fordividing the registration into the pre-processing phase and onlinephase, i.e., during a medical procedure, in accordance with someapplications of the present invention. For some applications, thepre-processing phases reduces the search space for the online phase.During the pre-processing phase, 3D image data of the skeletal portionis acquired (step 288), computer processor 22 is used to (i) generate N2D projection images from the 3D image data (step 290), (ii) determine aset of attributes that describe each of the 2D projection images, thenumber of attributes being smaller than the number of pixels in each 2Dprojection image (step 292), (iii) determine for each 2D projectionimage a respective value for each of the attributes (step 294), and (iv)store N respective sets of attributes, with respective values assignedfor each attribute, for the N 2D projection images (step 296). Thus,each data item in the original search space, in this case a DRRgenerated from the 3D scan data, is reduced in the pre-processing phaseinto a smaller number of characteristics, e.g., attributes. For someapplications, after storing the N respective sets of attributes, the N2D projection images are discarded.

During a medical procedure, i.e., in the online phase, only thosecharacteristics then need to be matched with an x-ray image in theonline phase, as follows: (i) a 2D radiographic image is acquired of theskeletal portion (step 298), (ii) computer processor 22 (a) determinesat least one specific set of values for the attributes that describe atleast a portion of the 2D radiographic image (step 300), (b) searchesamong the stored N respective sets of attributes for a set that bestmatches any of the at least one specific set of values (step 302), and(c) uses the set that best matches, to generate an additional 2Dprojection image from the 3D image data, the additional 2D projectionimage matching at least the portion of the 2D radiographic image (step304).

Reference is now made to FIG. 31, which is a flow chart for a method fordividing the registration into the pre-processing phase and onlinephase, i.e., during a medical procedure, in accordance with someapplications of the present invention. For some applications, theabove-described method is used for generating a first approximation andknown techniques may then be used for the final match. In step 306computer processor 22 (a) uses the set that best matches to generate aplurality of additional 2D projection images from the 3D image data,each of the plurality of additional projection images approximating atleast the portion of the 2D radiographic image, and (b) using theplurality of additional projection images, optimizes (step 308) to finda 2D projection image that matches at least the portion of the 2Dradiographic image.

For some applications, the pre-processing phase comprises the followingsteps (some of which are optional and the order of which may vary):

-   -   1. A targeted vertebra is marked upon the CT scan data by the        user.    -   2. An approximate center of the vertebra, or a point of interest        within the vertebra, is pointed at or calculated. It may also be        marked as part of the aforementioned pre-surgery planning.    -   3. Several sectors, each around a common imaging angle of the        x-ray that may be expected later on, during surgery (e.g., AP,        left lateral, right lateral, left oblique, right oblique), are        selected. For some applications, it may even be one sector        comprising an entire dome, or even an entire sphere.    -   4. The data points within each sector typically include x-ray        camera position in space, angles relative to the vertebra,        distance to the vertebra or to the selected point within the        vertebra, or any other applicable x-ray system parameter.    -   5. From each simulated x-ray system with its associated set of        parameters, a DRR of the vertebra is generated, such that        overall there are M DRRs.    -   6. Each DRR is presented as an N-dimensional vector, according        to a similarity measure involved in 3D-2D registration (it is        the same N for all DRRs). The coordinates of this vector are        calculated from grayscale values of the DRR pixels. These        calculations can include different image processing operations        such as filtering, convolutions, normalization and others.    -   7. If there were M DRRs, then there are now M points in the said        N-dimensional space.    -   8. Next, the M vectors are projected to a sub-space wherein the        sub-space has fewer than N dimensions, let's say D dimensions.        Typically, D is much smaller than N. One of the possible        techniques for generating such sub-space, also known as        Dimensionality Reduction techniques, is Principal Component        Analysis (PCA). Other known techniques that may be applied        include (See        https://en.wikipedia.org/wiki/Dimensionality_reduction)        Non-negative Matrix Factorization (NMF), or Kernel PCA, or        Graph-based kernel PCA, or Linear discriminant analysis (LDA),        or Generalized discriminant analysis (GDA), or any combination        thereof.

Typically, in the new D-dimensional sub-space, there are M vectors, eachcorresponding to one of the M DRRs. Each of the M vectors is now reducedto a point with D coordinates in the D-dimensional subspace.

Typically, from M N-dimensional vectors representing DRRs, there hasbeen a reduction to M points in a D-dimensional space. Therefore, theoutcome is a great reduction, by several orders of magnitude, the amountof data that we shall need to search in the next phase which is theonline phase.

For some applications, the online phase comprises the following steps(some of which being optional and the order of which may vary):

-   -   1. An x-ray image is acquired. A set of some or all of the        values of the applicable parameters related to the x-ray source        is: extracted from the display of the image, such as by means of        OCR or pattern recognition; indicated by the user; deduced from        analysis of the anatomy in the image; deduced from the        appearance of the radio-opaque markers in the image; read from        the DICOM file containing the image; received from the x-ray        system; or any combination thereof    -   2. According to those values of those parameters, the sub-space        corresponding to the same sector is searched, using the        aforementioned similarity measure. Due to the aforementioned        dimensionality reduction, the search can be done faster (by        orders of magnitude) compared with a situation where the        original N-dimensional space would have had to be searched.        Typically, during this search phase there is no need to        regenerate the DRRs that were generated in the pre-processing        phase.    -   3. As a result of the search, a point in the D-dimensional        subspace that best matches the current x-ray image is found.        Typically, the DRR from which this point was generated is        retrieved or re-generated and the x-ray image is co-registered        with the CT scan of the same vertebra to obtain an initial        approximation.    -   4. A fine-tuned 3D-2D co-registration follows. That is performed        using known techniques such as CMA-ES (covariance matrix        adaptation evolution strategy). A simulated x-ray source,        corresponding to the actual DRR best-matching the x-ray image,        is created.    -   5. If there is a singularity in the reconstruction in the CT        data of the tool that is detected in the x-ray image, it is        identified (typically automatically) and the user is prompted to        change the position of the x-ray source and re-acquire an x-ray        image. Examples for situations leading to a singularity include:        two x-ray images acquired in a such way that planes containing        the x-source and the tool projection on the x-ray detector        coincide for both acquisitions a single x-ray image acquired        where the tool is seen from a bull's-eye view.

For some applications, the steps of generating a plurality of DRRs froma 3D CT image, and identifying respective first and second DRRs thatmatch the 2D x-ray images of the vertebra are aided by deep-learningalgorithms.

For some applications, deep-learning techniques are performed as part ofthe processing of images of a subject's vertebra, as described in thefollowing paragraphs. By performing the deep-learning techniques, thesearch space for DRRs of the subject's vertebra that match the x-rayimages is limited, which reduces the intraprocedural processingrequirement, reduces the time taken to performing the matching, and/orreduces cases of dual solutions to the matching.

For some applications, deep learning may be performed using 3D scan dataonly of the targeted vertebra, which typically greatly facilitates thetask of building the deep-learning dataset. For some applications,during the deep-learning training phase, a large database of DRRsgenerated from the 3D data of the targeted vertebra, and (at least someof) their known parameters relative to vertebra, are inputted to adeep-learning engine. Such parameters typically include viewing angle,viewing distance, and optionally additional x-ray system and cameraparameters. For some applications, the aforementioned parameters areexact. Alternatively, the parameters are approximate parameters. Theparameters may be recorded originally when generating the DRRs, orannotated by a radiologist. Thus, the engine learns, given a certain 2Dprojection image, to suggest simulated camera and x-ray system viewingdistances and angles that correspond to that projection image.Subsequently, the deep-learning data is fed as an input to computerprocessor 22 of system 20. During surgery, in order to register any ofthe 2D x-ray images to the 3D image data, computer processor uses thedeep-learning data by inference in order to limit the search space inwhich DRRs of the 3D image data that match the x-ray images should besearched for. Computer processor 22 then searches for matching DRRs onlywithin the search space that was prescribed by the deep-learninginference.

The above-described registration steps are summarized in FIG. 18, whichis a flowchart showing steps that are performed by computer processor,in order to register 3D image data of a vertebra to two or more 2D x-rayimages of the vertebra.

In a first step 140, the search space for DRRs that match respectivex-ray images is limited, for example, using deep-learning data asdescribed hereinabove. Alternatively or additionally, in order to avoiddouble solutions when searching for a DRR that matches a given x-rayimage, the computer processor determines whether the x-ray images are,for example, AP, PA, left lateral, right lateral, left oblique, or rightoblique, and/or from which quadrant a tool is being inserted.

In step 141, a plurality of DRRs are generated within the search space.

In step 142, the plurality of DRRs are compared with the x-ray imagesfrom respective views of the vertebra.

In step 143, based upon the comparison, the DRR that best matches eachof the x-ray images of the vertebra is selected. Typically, for thesimulated camera position that would generate the best-matching DRR, thecomputer processor determines the viewing angle and viewing distance ofthe camera from the 3D image of the vertebra.

It is noted that the above steps are performed separately for each ofthe 2D x-ray images that is used for the registration. For someapplications, each time one or more new 2D x-ray images are acquired,the image(s) are automatically registered to the 3D image data using theabove described technique. The 2D to 3D registration is thereby updatedbased upon the new 2D x-ray acquisition(s).

Reference is now made to FIG. 19A, which is a flowchart showing steps ofan algorithm that is performed by computer processor 22 of system 20, inaccordance with some applications of the present invention.

As described hereinabove, for each of the x-ray images (denoted X1 andX2), the computer processor determines a corresponding DRR from asimulated camera view (the simulated cameras being denoted C1 for X1 andC2 for X2).

The 3D scan and two 2D images are now co-registered, and the following3D-2D bi-directional relationship generally exists:

-   -   Geometrically, a point P3D in the 3D scan of the body portion        (in three coordinates) is at the intersection in 3D space of two        straight lines    -   i. A line drawn from simulated camera C1 through the        corresponding point PX1 (in two image coordinates) in 2D image        X1.    -   ii. A line drawn from simulated camera C2 through the        corresponding point PX2 (in two image coordinates) in 2D image        X2.

Therefore, referring to FIG. 19A, for some applications, for a portionof a tool that is visible in the 2D images, such as the tool tip or adistal portion of the tool, the computer processor determines itslocation within the 3D image data (denoted TP3D), using the followingalgorithmic steps:

Step 145: Identify, by means of image processing, the tool's tip TPX1 inimage X1 (e.g., using the image processing techniques describedhereinabove). For some applications, to make the tool tip point betterdefined, the computer processor first generates a centerline for thetool and then the tool's distal tip TPX1 is located upon on thatcenterline.

In general, the computer processor identifies the locations of a tool ora portion thereof in the 2D x-ray images, typically, solely by means ofimage processing. For example, the computer processor may identify thetool by using a filter that detects pixel darkness (the tool typicallybeing dark), using a filter that detects a given shape (e.g., anelongated shape), and/or by using masks. For some applications, thecomputer processor compares a given region within the image to the sameregion within a prior image. In response to detecting a change in somepixels within the region, the computer processor identifies these pixelsas corresponding to a portion of the tool. For some applications, theaforementioned comparison is performed with respect to a region ofinterest in which the tool is likely to be inserted, which may be basedupon a known approach direction of the tool. For some applications, thecomputer processor identifies the portion of the tool in the 2D images,solely by means of image processing, using algorithmic steps asdescribed in US 2010-0161022 to Tolkowsky, which is incorporated hereinby reference. For some applications, the computer processor identifiesthe portion of the tool in the 2D images, solely by means of imageprocessing, using algorithmic steps as described in US 2012-0230565 toSteinberg, which is incorporated herein by reference. For someapplications, the tool or portion thereof is identified manually, andpointed at on one or more of the images, by the operator.

For some applications, identification of the portion of the tool in the2D images is facilitated, manually or automatically, by defining aregion of interest (ROI) in a 2D image around the planned insertion lineof the tool, as such line was determined in the planning phase usingtechniques described by the present application, and then registered tothe 2D image using techniques described by the present application.Next, the portion of the tool is searched within the ROI usingtechniques described by the present application.

Reference is made to FIGS. 19B-E, showing an example of the automaticdetection within an x-ray image, using the MatLab computing environment,of a tool that is inserted into a vertebra. FIG. 19B shows an x-rayimage in which a tool 310 may be observed. FIG. 19C shows an outcome ofactivating “vesselness” detection upon the x-ray image. FIG. 19D showsan outcome after applying a threshold to the vesselness measure. FIG.19E shows a line 312 representing the detected tool, which overlaps theactual tool, in the x-ray image.

Step 146: Generate a typically-straight line L1 from C1 to TPX1. (It isnoted that, as with other steps described as being performed by thecomputer processor, the generation of a line refers to a processing stepthat is the equivalent of drawing a line, and should not be construed asimplying that a physical line is drawn. Rather the line is generated asa processing step).Step 147: Identify, by means of image processing, the tool's tip TPX2 inimage X2 (e.g., using the image processing techniques describedhereinabove). For some applications, to make the tool tip point betterdefined, the computer processor first generates a centerline for thetool and then the tool's distal tip TPX2 is located upon on thatcenterline. The image processing techniques that are used to tool's tipTPX2 in image X2 are generally similar to those described above withreference to step 145.Step 148: Generate a typically-straight line L2 from C2 to TPX2.Step 149: Identify the intersection of L1 and L2 in 3D space as thelocation of the tool's tip relative to the 3D scan data.Step 150: Assuming that the shape of the tool is known (e.g., if thetool is a rigid or at least partially rigid tool, or if the tool can beassumed to have a given shape by virtue of having been placed intotissue), the computer processor derives the locations of additionalportions of the tool within 3D space. For example, in the case of a toolwith straight shaft in whole or in its distal portion, or one that maybe assumed to be straight once inserted into bone, or at least straightin its distal portion once inserted into bone, then this shaft, or atleast its distal portion, resides at the intersection of two planes,each extending from the simulated camera to the shaft (or portionthereof) in the corresponding 2D image. For some applications, thedirection of the shaft from its tip to proximal and along theintersection of the two planes is determined by selecting a pointproximally to the tool's tip on any of the x-ray images and observingwhere a line generated between such point and the correspondingsimulated camera intersects the line of intersection between the twoplanes.

It is noted that, since the co-registration of the 3D image data to the2D images is bidirectional, for some applications, the computerprocessor identifies features that are identifiable within the 3D imagedata, and determines the locations of such features with respect to the2D x-rays, as described in further detail hereinbelow. The locations ofeach such feature with respect to any of the 2D x-rays are typicallydetermined by (a) generating a typically-straight line from thesimulated camera that was used to generate the DRR corresponding to suchx-ray image and through the feature within the 3D image data and (b)thereby determining the locations of the feature with respect to thex-ray images themselves. For some applications, the locations of suchfeatures with respect to the 2D x-ray images are determined bydetermining the locations of the features within the DRRs that match therespective x-ray images, and assuming that the features will be atcorresponding locations within the matching x-ray images.

For some applications, based upon the registration, 3D image data isoverlaid upon a 2D image. However, typically, the 3D image data (e.g., a3D image, a 2D cross-section derived from 3D image data, and/or a 2Dprojection image derived from 3D image data) are displayed alongside 2Dimages, as described in further detail hereinbelow.

Reference is now made to FIG. 20A, which shows an example ofcross-sections 160 and 162 of a vertebra corresponding, respectively, tofirst and second locations of a tip 164 of a tool that is advanced intothe vertebra along a longitudinal insertion path, as shown oncorresponding 2D x-ray images, in accordance with some applications ofthe present invention. Typically, the tool has a straight shaft in wholeor in its distal portion, and/or may be assumed to be straight onceinserted into bone, or at least straight in its distal portion onceinserted into bone. Referring also to step 84 of FIG. 6, for someapplications, based upon the identified location of the tip of tool withrespect to one or more 2D x-ray image of the vertebra that are acquiredfrom a single image view, and the registration of an x-ray from thesingle 2D x-ray image view to the 3D image data (e.g., by matching a DRRfrom the 3D image data to the 2D x-ray image), computer processor 22determines a location of the tip of the tool with respect to a DRR thatis derived from the 3D image data (e.g., the DRR that was determined tomatch the 2D x-ray image), and in response thereto, drives the displayto display a cross-section of the vertebra, the cross-section beingderived from the 3D image data, and corresponding to the location of thetool tip. The cross-section is typically of a given plane at theidentified location. Typically, the cross-section is an axialcross-section, but for some applications, the cross-section is asagittal cross-section, a coronal cross-section, and/or a cross-sectionthat is perpendicular to or parallel with the direction of the toolinsertion.

For some applications, upon the cross-section, the computer processordrives the display to show a line 166 (e.g., a vertical line),indicating that the location of the tip of the tool is somewhere alongthat line. For some applications, the line is drawn vertically upon anaxial cross-section of the vertebra, as shown. For some applications,the surgeon is able to determine the likely location of the tool alongthe line based upon their tactile feel. Alternatively or additionally,based on the 3D image data, the computer processor drives the display todisplay how deep below the skin the vertebra is disposed, which acts asa further aid to the surgeon in determining the location of the toolalong the line.

As noted above, typically it is possible to generate an output as shownin FIG. 20A, by acquiring one or more 2D x-ray images from only a singlex-ray image view of the tool and the vertebra, and registering one ofthe 2D x-ray images to the 3D image data using the registrationtechniques described herein. Typically, by registering the 2D x-rayimage acquired from the single image view to the 3D image data, computerprocessor 22 determines, with respect to 3D image data (e.g., withrespect to the DRR that was determined to match the 2D x-ray image), (a)a plane in which the tip of the tool is disposed, and (b) a line withinthe plane, somewhere along which the tip of the tool is disposed, asshown in FIG. 20A. As described hereinabove, typically, when the tip ofthe tool is disposed at an additional location with respect to thevertebra, further 2D x-ray images of the tool at the additional locationare acquired from the same single x-ray image view, or a differentsingle x-ray image view, and the above-described steps are repeated.Typically, for each location of the tip of the tool to which theabove-described technique is applied, 2D x-ray images need only beacquired from a single x-ray image view, which may stay the same for therespective locations of the tip of the tool, or may differ forrespective locations of the tip of the tool.

Reference is now made to FIG. 20B, which is a schematic illustration ofthe location of the tool tip 168 denoted by cross-hairs uponcross-sections 160 and 162 of the vertebra corresponding, respectively,to first and second locations of a tip 164 of a tool that is advancedinto the vertebra along a longitudinal insertion path (as shown in FIG.15A), in accordance with some applications of the present invention. Forsome applications, by initially registering two or more 2D x-ray imagesof the tool and the vertebra that were acquired from respective 2D x-rayimage views, to the 3D image data, the precise location of the tip ofthe tool within a cross-section derived from the 3D image data isdetermined and indicated upon the cross-section, as shown in FIG. 20A.As described hereinbelow, with reference to FIGS. 25A-25B, for someapplications, after initially determining the location of the tip of thetool with respect to the 3D image data using two or more 2D x-ray imagesof the tool and the vertebra that were acquired from respective 2D x-rayimage views, subsequent locations of the tip of the tool are determinedwith respect to the 3D image data by acquiring further x-ray images fromonly a single x-ray image view.

Reference is now made to FIGS. 21A and 21B, which show examples of adisplay showing a relationship between an anticipated longitudinalinsertion path 170 of a tool 172 and a designated location 174 upon,respectively, AP and lateral 2D x-ray images, in accordance with someapplications of the present invention. Reference is also made to step 86of FIG. 6.

For some applications, a location within a vertebra is designated withinthe 3D image data. For example, an operator may designate a targetportion (e.g. a fracture, a tumor, a virtual pedicle screw, etc.),and/or a region which the tool should avoid (such as the spinal cord)upon the 3D image data (e.g., a 3D image, a 2D cross-section derivedfrom 3D image data, and/or a 2D projection image derived from 3D imagedata). Alternatively or additionally, the computer processor mayidentify such a location automatically, e.g., by identifying the portionvia image processing. Based upon the registration of the first andsecond 2D x-ray images to the 3D image data, the computer processorderives a position of the designated location within at least one of thex-ray images, using the techniques described hereinabove. In addition,the computer processor determines an anticipated path of the tool withinthe x-ray image. Typically, the computer processor determines theanticipated path by determining a direction of an elongate portion ofthe tool (and/or a center line of the elongate portion) within the x-rayimage. Since the tool is typically advanced along a longitudinalinsertion path, the computer processor extrapolates the anticipated pathby extrapolating a straight line along the determined direction.

For some applications, the computer processor performs a generallysimilar process, but with respect to a desired approach vector (e.g.,for insertion and implantation of a screw) that, for example, is inputinto the computer processor manually, and/or is automatically derived bythe processor. For example, such an approach vector may have beengenerated during a planning phase, typically upon the 3D image data, andbased upon the insertion of a simulated tool into the vertebra.Typically, such an approach vector is one that reaches a desired target,while avoiding the spinal cord or exiting the vertebra sideways.

For some applications, in response to the above steps, the computerprocessor generates an output indicating a relationship between theanticipated longitudinal insertion path of the tool and the designatedlocation. For some applications, the computer processor generates anoutput on the display, e.g., as shown in FIGS. 21A and 21B.Alternatively or additionally, the computer processor may generateinstructions to the operator to redirect the tool. Further alternativelyor additionally, the computer processor may generate an alert (e.g., anaudio or visual alert) in response to detecting that the tool isanticipated to enter a region that should be avoided (such as the spinalcord) or is anticipated to exit the vertebra sideways in the otherdirection.

Referring again to step 90 of FIG. 6, for some applications, computerprocessor 22 determines a location of a portion of the tool with respectto the vertebra, within the x-ray images, by means of image processing,as described hereinabove. Based upon the identified location of theportion of the tool within the x-ray images, and the registration of thefirst and second 2D x-ray images to the 3D image data, the computerprocessor determines the location of the portion of the tool withrespect to the 3D image data. For some applications, in responsethereto, the computer processor shows an image of the tool itself, or asymbolic representation thereof, overlaid upon the 3D image data.Alternatively or additionally, the computer processor derives arelationship between the location of the portion of the tool withrespect to the 3D image data and a given location within the 3D imagedata, and generates an output that is indicative of the relationship. Asdescribed hereinabove, the registration of the 2D images to the 3D imagedata is typically performed with respect to individual vertebrae.Therefore, even is the subject has moved between the acquisition of the3D image data and the acquisitions of the 2D images, the techniquesdescribed herein are typically effective.

For some applications, the representation of the actual tool (or of aportion thereof) is displayed relative to the planned path of insertion,in accordance with techniques described by the present application. Forsome applications, the planned path of insertion is generated byembodiments of the present invention. For some applications, the actualtool vs. the planned path is displayed upon a 2D slice or a 2Dprojection of the 3D data. For some applications, the actual tool vs.the planned path is displayed upon a 3D model generated from the 3Ddata, with such model typically having some level of transparencyallowing to see the representations within it. For some applications,the 3D model is auto-rotated to facilitate the operator's spatialcomprehension of actual tool vs. planned path. For some applications,the actual tool vs. the planned path is displayed upon a 2D x-ray imagein which the tool can be observed and with the planned path registeredfrom the 3D data, for example by means of matching a DRR generated fromthe 3D data and comprising the planned path with the 2D x-ray image. Forsome applications, the planned path comprises one or more points alongthe path, such as the incision site at skin level, the entry point intothe vertebra, the out-of-pedicle point, and the target point, or anycombination thereof.

Reference is made to FIG. 21C, which shows an example of therepresentations of a portion of an actual tool 314 (solid line) and theplanned insertion path 316 (dashed line) displayed within asemi-transparent 3D model of a spinal segment, in accordance with someapplications of the present invention.

For some applications, the computer processor generates an output thatis indicative of the distance of the tip of the tool from the spinalcord and/or outer vertebral border, e.g., using numbers or colorsdisplayed with respect to the 3D image data. For some applications, thecomputer processor outputs instructions (e.g., textual, graphical, oraudio instructions) indicating that the tool should be redirected. Forsome applications, as an input to this process, the computer processordetermines or receives a manual input indicative of a direction ororientation from which the tool is inserted (e.g., from top or bottom,or left or right).

Reference is now made to FIG. 22A, which shows an AP x-ray of two tools176L and 176R being inserted into a vertebra through, respectively,10-11 o'clock and 1-2 o'clock insertion windows, and to FIG. 22B, whichshows a corresponding lateral x-ray image to FIG. 22A, the images beingacquired in accordance with prior art techniques. As describedhereinabove, in many cases, during spinal surgery, two or more tools areinserted into a vertebra, for example, from the 10 o'clock to 11 o'clockinsertion window and from the 1 o'clock to 2 o'clock insertion window,with the process repeated, as applicable, for one or more furthervertebrae. Within the AP x-ray view, tools 176L and 176R, inserted intorespective windows, are typically discernible from one another, as shownin FIG. 22A.

Furthermore, with reference to FIGS. 5A-B, for some applications, withinthe AP view, the computer processor discerns between tool inserted viathe respective insertion windows based upon the arrangements of markersets 50. However, if the tools are of identical or similar appearance,then from some imaging directions it is challenging to identify whichtool is which. In particular, it is challenging to identify which toolis which in lateral x-ray views, as may be observed in FIG. 22B. Ingeneral, it is possible to discern between tools in images acquiredalong the direction of insertion, and more difficult to discern betweentools in images acquired along other directions.

Reference is now made to FIGS. 23A-B, which show flowcharts for matchingbetween a tool in one x-ray image acquired from a first view, and thesame tool in a second x-ray image acquired from a second view. For someapplications, computer processor 22 matches automatically between a toolin one x-ray image acquired from a first view, and the same tool in asecond x-ray image acquired from a second view, using techniquesdescribed by the present application and comprising the following steps:

(i) acquiring 3D image data of a skeletal portion (step 318),

(ii) planning respective longitudinal insertion paths for each of atleast two tools (step 320),

(iii) associating the planned respective longitudinal insertion pathswith the 3D image data (step 322),

(iv) while respective portions of the tools are disposed at firstrespective locations along their respective longitudinal insertion pathswith respect to the skeletal portion, acquiring two 2D x-ray images ofthe skeletal portion from two different respective image views (step324) (typically using an x-ray imaging device that is not registeredwith respect to the subject's body), and

(v) using computer processor 22, automatically matching between a toolin the first 2D x-ray image and the same tool in the second 2D x-rayimage (step 326).

FIG. 23B shows the steps computer processor 22 performs to do theautomatic matching, as follows:

(A) identifying respective tool elements of each of the tools withineach of the first and second 2D x-ray images, by means of imageprocessing (step 328),

(B) registering the first and second x-ray images to the 3D image data,as described hereinabove (step 330), and

(C) based upon the identified respective tool elements within the firstand second 2D x-ray images, and the registration of the first and second2D x-ray images to the 3D image data, identifying for at least one toolelement within the first and second 2D x-ray images a correspondencebetween the tool element and the respective planned longitudinalinsertion path for that tool (step 332), i.e., the planned insertionline of each tool is matched with the tool observed in the x-ray imagesto be nearest that line, after the planning data from the CT has beenprojected (i.e., overlaid) onto the x-ray image.

Once a correspondence is made in both the first and second x-raysbetween a tool element in the x-rays and its corresponding plannedlongitudinal insertion path, computer processor 22 thus identifies whichtool in the first x-ray is the same tool in the second x-ray and canthen position respective representations of the respective tool elementswithin a display of the 3D image data.

For some applications, the computer processor matches automaticallybetween a tool in one x-ray image acquired from a first view, and thesame tool in a second x-ray image acquired from a second view, bydefining a region of interest (ROI) in each x-ray image around theplanned insertion line of the tool, as such line was determined in theplanning phase using techniques described by the present application andthen registered to the 2D image using techniques described by thepresent application, and then matching between instances of the tool, orportions thereof, that appear in both ROIs.

For some applications, the planned insertion line of each tool isdisplayed distinctively, e.g., each in a unique color within the 3Dimage data. The planned respective longitudinal insertion paths may alsobe distinctively overlaid on the first and second x-ray images,facilitating identification of each insertion path in the x-ray imageson which the planning data has been projected (i.e., overlaid), and thusfacilitating manual association of each tool with a nearby plannedinsertion line, e.g., how close the tool is to the planned insertionline, in each of the x-ray images and for each tool among the x-rayimages.

For some applications, the planning data (or portions thereof) is, usingtechniques described by the present application, projected and displayedupon each x-ray image that is acquired and registered with the 3D data.For some applications, a first tool (e.g., needle, wire) seen in anx-ray image is distinguished, typically automatically and typically bemeans of image processing, from a second tool (e.g., forceps) used tograb the first tool, by the first tool having a single longitudinalshaft and the second tool having a dual longitudinal shaft.

Referring again to step 90 of FIG. 6, for some applications, rather thandisplaying the tool, a representation thereof, and/or a path thereofupon a 3D image, the computer processor drives the display to displaythe tool, a representation thereof, and/or a path thereof upon a 2Dcross-section of the vertebra that is derived from the 3D image. Forsome applications, the computer processor determines the location of thecenterline of the tool shaft, by means of image processing. For example,the computer processor may use techniques for automatically identifyinga centerline of an object as described in US 2010-0161022 to Tolkowsky,which is incorporated herein by reference. For some applications, thecomputer processor drives the display to display the centerline of thetool upon the 3D image data, the end of the centerline indicating thelocation of the tool tip within the 3D image data. Alternatively oradditionally, the computer processor drives the display to display anextrapolation of the centerline of the tool upon the 3D image data, theextrapolation of the centerline indicating an anticipated path of thetool with respect to the 3D image data. For some applications, thecomputer processor drives the display to display a dot at the end of theextrapolated centerline upon the 3D image data, the dot representing theanticipated location of the tip of the tool.

For some applications, the computer processor drives the display todisplay in a semi-transparent format a 3D image of the vertebra with thetool, a representation thereof, and/or a path thereof disposed insidethe 3D image. Alternatively or additionally, the computer processordrives the display to rotate the 3D image of the vertebra automatically(e.g., to rotate the 3D image back-and-forth through approximately 30degrees). For some applications, the computer processor retrieves animage of a tool of the type that is being inserted from a library andoverlays the image upon the derived centerline upon the 3D image data.Typically, the tool is placed along the centerline at an appropriatescale with the dimensions being derived from the 3D image data. For someapplications, a cylindrical representation of the tool is overlaid uponthe derived centerline upon the 3D image data. For some applications,any one of the above representations is displayed relative to apredesignated tool path, as derived automatically by processor 22, or asinput manually by the surgeon during a planning stage.

Referring again to FIG. 2, tool insertion into a vertebra must avoid thespinal cord 42, and at the same time needs to avoid exiting the vertebrafrom the sides, leaving only two narrow tool insertion windows 44, oneither side of the vertebra. Typically, the greater the level ofprotrusion of a tool or implant into the spinal cord, the worse theclinical implications. For some applications, volumes within the 3Dimage of the vertebra (and/or a cross-sectional image derived therefrom)are color coded (e.g., highlighted) to indicate the level ofacceptability (or unacceptability) of protrusion into those volumes. Forsome applications, during the procedure, the computer processordetermines the location of the tool with respect to the 3D image data,and in response thereto, the computer processor drives the display tohighlight a vertebral volume into which there is a protrusion that isunacceptable. For some applications, the computer processor drives thedisplay to display a plurality (e.g., 2-6) of, typically concentric,cylinders within the 3D image of the vertebra, the cylinders indicatingrespective levels of acceptability of protrusion of a tool into thevolumes defined by the cylinders. During the procedure, the computerprocessor determines the location of the tool with respect to the 3Dimage data, and in response thereto, the computer processor drives thedisplay to highlight the cylinder in which the tool or a portion thereofis disposed, and/or is anticipated to enter. For some applications, thecomputer processor performs the above-described functionalities, but notwith respect to the tool that is currently being inserted (which may bea narrow tool, such as a needle), rather with respect to the eventualimplant (e.g., a pedicle screw, which typically has a larger diameter)that will be positioned later using the current tool. For someapplications, the computer processor performs the above-described stepswith respect to a 2D cross-sectional image that is derived from the 3Dimage data. For such cases, rectangles, rather than cylinders aretypically used to represent the respective levels of acceptability ofprotrusion of a tool into the areas defined by the rectangles.

For some applications, the processor allows a 3D image of the vertebrawith the tool, a representation of the tool, and/or a path of the toolindicated within the image to be rotated, or the processor rotates theimage automatically, in order for the user to better understand the 3Dplacement of the tool. It is noted that, since the images of thevertebra and the tool were input from different imaging sources, thesegmented data of what is the tool (or its representation) and what isthe vertebra is in-built (i.e., it is already known to the computerprocessor). For some applications, the computer processor utilizes thisin-built segmentation to allow the operator to virtually manipulate thetools with respect to the vertebra. For example, the operator mayvirtually advance the tool further along its insertion path, or retractthe tool and observe the motion of the tool with respect to thevertebra. For some applications, the computer processor automaticallyvirtually advances the tool further along its insertion path, orretracts the tool with respect to the vertebra in the 3D image data.

For some applications, accuracy of determining the position of theportion of the tool within the 3D image data is enhanced by registeringthree 2D x-ray images to the 3D image data, the images being acquiredfrom respective, different views from one another. Typically, for suchapplications, an oblique x-ray image view is used in addition to AP andlateral views. For some applications, accuracy of determining theposition of the portion of the tool within the 3D image data is enhancedby using x-ray images in which multiple portions of the tool, orportions of multiple tools, are visible and discernible from one anotherin the x-ray images. For some applications, the tools are discerned fromone another based on a manual input by the operator, or automatically bythe computer processor. For some applications, accuracy of determiningthe position of the portion of the tool within the 3D image data isenhanced by referencing the known shapes and/or dimensions of radiopaquemarkers 52 as described hereinabove.

Reference is now made to FIG. 24, which is a schematic illustration ofJamshidi™ needle 36 with a radiopaque clip 180 attached thereto, inaccordance with some applications of the present invention. For someapplications, accuracy of determining the position of the portion of thetool within the 3D image data is enhanced by adding an additionalradiopaque element to the tool (such as clip 180), such that the toolhas at least two identifiable features in each 2D image, namely, itsdistal tip and the additional radiopaque element. For some applications,the additional radiopaque element is configured to be have a defined 3Darrangement such that the additional radiopaque element providescomprehension of the orientation of the tool. For example, theadditional radiopaque element may include an array of radiopaquespheres. For some applications, the additional radiopaque elementfacilitates additional functionalities, e.g., as described hereinbelow.For some applications, the tool itself includes more than one radiopaquefeature that is identifiable in each 2D x-ray image. For suchapplications, an additional radiopaque element (such as clip 180) istypically not attached to the tool.

For some applications, the imaging functionalities described above withreference to the 3D image data are performed with respect to the 2Dx-ray images, based upon the co-registration of the 2D images to the 3Dimage data. For example, the tool may be color-coded in the x-ray imagesaccording to how well the tool is placed. For some applications, if thetool is placed incorrectly, the computer processor drives the display toshow how the tool should appear when properly placed, within the 2Dx-ray images.

Reference is now made to FIGS. 25A and 25B, which show examples of APx-ray images and corresponding lateral x-ray images of a vertebra, atrespective stages of the insertion of a tool into the vertebra, inaccordance with some applications of the present invention. Reference isalso made to step 88 of FIG. 6. A common practice in spinal surgery thatis performed under x-ray is to use two separate c-arm poses (typicallyany two of AP, lateral and oblique) to gain partial 3D comprehensionduring tool insertion and/or manipulation. This typically requiresmoving the C-arm back and forth, and exposes the patient to a highradiation dose.

For some applications of the present invention, images are initiallyacquired from two poses, which correspond to respective image views. Forexample, FIG. 25A shows examples of AP and lateral x-ray images of atool being inserted dorsally into a vertebra. Subsequently, the C-arm ismaintained at a single pose for repeat acquisitions during toolinsertion and/or manipulation, but the computer processor derives theposition of the tool with respect to the vertebra in additional x-rayimaging views, and drives the display to display the derived position ofthe tool with respect to the vertebra in the additional x-ray imageviews. For example, FIG. 19B shows an example of an AP image of the tooland the vertebra of FIG. 25A, but with the tool having advanced furtherinto the vertebra relative to FIG. 2A. Based upon the AP image in whichthe tool has advanced the computer processor derives the new, calculatedposition of the tool with respect to the lateral x-ray imaging view, anddrives the display to display a representation 190 of the new toolposition upon the lateral image. Typically, the new, calculated toolposition is displayed upon the lateral image, in addition to thepreviously-imaged position of the tool tip within the lateral image, asshown in FIG. 25B. Typically, the computer processor derives thelocation of portion of the tool with respect to one of the two original2D x-ray image views, based upon the current location of the portion ofthe tool as identified within a current 2D x-ray image, and arelationship that is determined between images that were acquired fromthe two original 2D x-ray image views, as described in further detailhereinbelow.

For some applications, the repeat acquisitions are performed from a 2Dx-ray image view that is the same as one of the original 2D x-ray imageviews, while for some applications the repeat acquisitions are performedfrom a 2D x-ray image view that is different from both of the original2D x-ray image views. For some applications, in the subsequent step, thetool within the vertebra is still imaged periodically from one or moreadditional 2D x-ray image views, in order to verify the accuracy of theposition of the tool within the additional views that was derived by thecomputer processor, and to correct the positioning of the tool withinthe additional 2D x-ray image views if necessary. For some applications,the C-arm is maintained at a single pose (e.g., AP) for repeatacquisitions during tool insertion and/or manipulation, and the computerprocessor automatically derives the location of portion of the tool withrespect to the 3D image data of the vertebra, and updates the image ofthe tool (or a representation thereof) within the 3D image data.

Typically, applications as described with reference to FIGS. 25A-B areused with a tool that is inserted into the skeletal anatomy along alongitudinal (i.e., a straight-line, or generally-straight-line)insertion path. For some applications, the techniques are used with atool that is not inserted into the skeletal anatomy along astraight-line insertion path. For such cases, the computer processortypically determines the non-straight line anticipated path of progressof the tool by analyzing prior progress of the tool, and/or by observinganatomical constraints along the tool insertion path and predictingtheir effect. For such applications, the algorithms describedhereinbelow are modified accordingly.

For some applications, the techniques described with reference to FIGS.25A-B are performed with respect to a primary x-ray imaging view whichis typically from the direction along which the intervention isperformed (and typically sets 50 of markers 52 are placed on or near thesubject such that the markers appear in this imaging view), and asecondary direction from which images are acquired to provide additional3D comprehension. In cases in which interventions are performeddorsally, the primary x-ray imaging view is typically generally AP,while the secondary view is typically generally lateral.

For some applications, computer processor 22 uses one of the followingalgorithms to perform the techniques described with reference to FIGS.25A-B.

Algorithm 1:

-   -   1. The original two 2D x-ray images X1 and X2 are registered to        3D image data using the techniques described hereinabove.    -   2. Based upon the registration, a generally-straight-line of the        tool TL (e.g., the centerline, or tool shaft), as derived from        the 2D x-ray images, is positioned with respect to the 3D image        data as TL-3D.    -   3. The generally-straight-line of the tool with respect to the        3D image data is extrapolated to generate a forward line F-TL3D        with respect to the 3D image data.    -   4. When the tool is advanced, a new 2D x-ray X1{circumflex over        ( )} is acquired from one of the prior poses only, e.g., from        the same pose from which the original X1 was acquired.        (Typically, to avoid moving the C-arm, this is the pose at which        the most recent of the two previous 2D x-rays was acquired.)        -   For some applications, the computer processor verifies that            there has been no motion of the C-arm with respect to the            subject, and/or vice versa, between the acquisitions of X1            and X1{circumflex over ( )}, by comparing the appearance of            markers 52 in the two images. For some applications, if            there has been movement, then Algorithm 2 described            hereinbelow is used.    -   5. The computer processor identifies, by means of image        processing, the location of the tool's distal tip in image        X1{circumflex over ( )}. This is denoted TPX1{circumflex over        ( )}.    -   6. The computer processor registers 2D x-ray image X1{circumflex        over ( )} to the 3D image data using the DRR that matches the        first x-ray view. It is noted that since pose did not change        between the acquisitions of X1 and X1{circumflex over ( )}, the        DRR that matches x-ray X1{circumflex over ( )} is same as for        x-ray X1. Therefore, there is no need to re-search for the best        DRR to match to x-ray X1{circumflex over ( )}.    -   7. The computer processor draws a line with respect to the 3D        image data from C1 through TPX1{circumflex over ( )}.    -   8. The computer processor identifies the intersection of that        line with the F-TL3D line as the new location of the tip, with        respect to the 3D image data. It is noted that in cases in which        the tool has been retracted, the computer processor identifies        the intersection of the line with the straight-line of the tool        with respect to the 3D image data TL-3D, rather than with        forward line F-TL3D with respect to the 3D image data.    -   9. The computer processor drives the display to display the tool        tip (or a representation thereof) at its new location with        respect to the 3D image data, or with respect to x-ray image X2.        Algorithm 2:    -   1. The original two 2D x-ray images X1 and X2 are registered to        3D image data using the techniques described hereinabove.    -   2. Based upon the registration, a generally-straight-line TL of        the tool (e.g., the centerline, or tool shaft) as derived from        the x-ray images is positioned with respect to the 3D image data        as TL-3D.    -   3. The generally-straight-line of the tool with respect to the        3D image data is extrapolated to generate a forward line F-TL3D        with respect to the 3D image data.    -   4. When the tool is advanced, a new 2D x-ray X3 is acquired        from, typically, any pose, and not necessarily one of the prior        two poses.    -   5. The computer processor identifies, by means of image        processing, the location of the tool's distal tip in image X3.        This is denoted TPX3.    -   6. The computer processor registers 2D x-ray image X3 to the 3D        image data of the vertebra by finding a DRR that best matches 2D        x-ray image X3, using the techniques described hereinabove. The        new DRR has a corresponding simulated camera position C3.    -   7. The computer processor draws a line with respect to the 3D        image data from C3 through TPX3.    -   8. The computer processor identifies the intersection of that        line with the F-TL3D line as the new location of the tip, with        respect to the 3D image data. It is noted that in cases in which        the tool has been retracted, the computer processor identifies        the intersection of the line with the straight-line of the tool        with respect to the 3D image data TL-3D, rather than with        forward line F-TL3D with respect to the 3D image data.    -   9. The computer processor drives the display to display the tool        tip (or a representation thereof) at its new location with        respect to the 3D image data, or with respect to x-ray image X1        and/or X2.

For some applications, Algorithm 1 or Algorithm 2 are furtherfacilitated by adding a radio-opaque feature, for example by means ofclipping, typically to the out-of-body portion of the tool. In suchcases, a feature, or an identifiable sub-feature thereof, serves as asecond feature, in addition to the tool's distal tip, for determiningthe direction of the tool's shaft. For some applications, the clip, oranother radiopaque feature attached to the tool, are as shown in FIG.24. For some applications, the clip, or another radiopaque featureattached to the tool, improve the accuracy of determining the directionof the tool's shaft.

For some applications, for Algorithm 1 or Algorithm 2, a softwarealgorithm is applied for identifying situations of singularity, withrespect to the tool, of X-Ray images X1 and X2. For some applications,such algorithm not only identifies the singularity but also recommendswhich of X1 and/or X2 should be reacquired from a somewhat differentpose. For some applications, such algorithm also guides the user as towhat such new pose may be. For some applications, the aforementionedclip, or another radiopaque feature attached to the tool, assists inidentifying and/or resolving situations of singularity between x-rayimages X1 and X2.

For some applications, the use of Algorithm 1 or Algorithm 2 has anadditional benefit of reducing the importance that the X-ray images areacquired in what is known as Ferguson views. In a Ferguson view, the endplates appear as flat and as parallel to one another as possible. It isconsidered advantageous for proper tool insertion into a vertebra.However, once any acquired 2D x-ray image is co-registered with the 3DCT data, as described by applications of the present invention, andfurthermore once a tool seen in the 2D x-ray images is registered withthe 3D data, again as described by applications of the presentinvention, the operator can assess in 3D the correctness of theinsertion angle and without needing x-ray images acquired specificallyin Ferguson view. Typically, it takes multiple trials-and-errors, whenmanipulating an x-ray c-arm relative to the subject's body, to achieveFerguson views. Multiple x-ray images are typically acquired in theprocess till the desired Ferguson view is achieved, with potentialadverse implications on procedure time and the amount of radiation towhich the subject and medical staff who are present are exposed.

For some applications, the use of Algorithm 1 or Algorithm 2 has anadditional benefit of reducing the importance that the X-ray images areacquired in what is known as “bull's-eye” views. In a “bull's-eye” view,the tool being inserted is viewed from the direction of insertion,ideally with the tool seen only as a cross-section, to furtherfacilitate the surgeon's understanding of where the tool is headedrelative to the anatomy. However, once any acquired 2D x-ray image isco-registered with the 3D CT data, as described by applications of thepresent invention, and furthermore once a tool seen in the 2D x-rayimages is registered with the 3D data, again as described byapplications of the present invention, the operator can assess in 3D thecorrectness of the insertion angle and without needing x-ray imagesacquired specifically in “bull's-eye” view. Typically, it takes multipletrials-and-errors, when manipulating an x-ray c-arm relative to thesubject's body, to achieve “bull's-eye” views. Multiple x-ray images aretypically acquired in the process till the desired “bull's-eye” view isachieved, with potential adverse implications on procedure time and theamount of radiation to which the subject and medical staff who arepresent are exposed.

For some applications of the present invention, the operator is assistedin manipulating the c-arm to a Ferguson view prior to activating thec-arm for acquiring images. On the system's display, the vertebra in 3D,with the tool depicted upon it, is rotated to a Ferguson view. Next, theoperator manipulates the c-arm such that the tool is positioned relativeto the detector at a similar angle to the one depicted on the system'sdisplay relative to the operator; only then is the c-arm activated toacquire x-ray images.

Algorithm 3:

Reference is now made to FIG. 26, which is a schematic illustration of athree-dimensional rigid jig 194 that comprises at least portions 196thereof that are radiopaque and function as radiopaque markers, theradiopaque markers being disposed in a predefined three-dimensionalarrangement, in accordance with some applications of the presentinvention. For some applications, as shown, radiopaque portions 196 areradiopaque spheres (which, for some applications, have different sizesto each other, as shown), and the spheres are coupled to one another byarms 198 that are typically radiolucent. Typically, the spheres arecoupled to one another via the arms, such that the spatial relationshipsbetween the spheres are known precisely.

The following algorithm is typically implemented by computer processor22 even in cases in which the x-ray images are not registered with 3Dimage data of the vertebra. Typically, this algorithm is for use with athree-dimensional radiopaque jig, such as jig 194, sufficient portionsof which are visible in all applicable x-ray images and can be used torelate them to one another. For some applications, the jig includes a 3Darray of radiopaque spheres, as shown in FIG. 20. For example, the jigmay be attached to the surgical table.

-   -   1. The original two 2D x-ray images X1 and X2 are registered to        one another, using markers of the jig as an anchor to provide a        3D reference frame.    -   2. When the tool is advanced, a new x-ray X1{circumflex over        ( )} is acquired from one of the prior poses, e.g., from the        same pose from which the original X1 was acquired. (Typically,        to avoid moving the C-arm, this is the pose at which the most        recent of the two-previous x-ray was acquired.)        -   For some applications, the computer processor verifies that            there has been no motion of the C-arm with respect to the            subject, and/or vice versa, between the acquisitions of X1            and X1{circumflex over ( )}, by comparing the appearance of            markers 52 (typically, relative to the subject's visible            skeletal portion), and/or portions 196 of jig 194            (typically, relative to the subject's visible skeletal            portion), in the two images. For some applications, if there            has been movement, then one of the other algorithms            described herein is used.    -   3. The computer processor identifies, by means of image        processing, the location of the tool's distal tip in image        X1{circumflex over ( )}. This is denoted TPX1{circumflex over        ( )}.    -   4. The computer processor registers 2D x-ray image X1{circumflex        over ( )} with X2 using the jig.    -   5. The computer processor calculates the new location of the        tool tip upon X2, based upon the registration.    -   6. The computer processor drives the display to display the tool        tip (or a representation thereof) at its new location with        respect to x-ray image X2.        Algorithm 4:

The following algorithm is typically implemented by computer processor22 even in cases in which the x-ray images are not registered with 3Dimage data of the vertebra. Typically, this algorithm is for use with atool that has two or more identifiable points in each 2D x-ray image.For example, this algorithm may be used with a tool to which a clip, oranother radiopaque feature is attached as shown in FIG. 24.

-   -   1. Within the original two 2D x-ray images X1 and X2, the        computer processor identifies, by means of image processing, the        two identifiable points of the tool, e.g., the distal tip and        the clip.    -   2. The computer processor determines a relationship between X1        and X2, in terms of image pixels. For example:        -   a. In X1, the two-dimensional distances between the tool tip            and the clip are dx1 pixels horizontally and dy1 pixels            vertically.        -   b. In X2, the two-dimensional distances between the tool tip            and the clip are dx2 pixels horizontally and dy2 pixels            vertically        -   c. Thus, the computer processor determines a 2D relationship            between the two images based upon the ratios dx2:dx1 and            dy2:dy1.    -   3. When the tool is advanced, a new x-ray X1{circumflex over        ( )} is acquired from one of the prior poses, e.g., from the        same pose from which the original x-ray X1 was acquired.        (Typically, to avoid moving the C-arm, this will be the pose at        which the most recent of the previous x-rays was acquired.)        -   For some applications, the computer processor verifies that            there has been no motion of the C-arm with respect to the            subject, and/or vice versa, between the acquisitions of X1            and X1{circumflex over ( )}, by comparing the appearance of            markers 52 in the two images. For some applications, if            there has been movement, then one of the other algorithms            described herein is used.    -   4. The computer processor identifies, by means of image        processing, the tip of the tool in image X1{circumflex over        ( )}.    -   5. The computer processor determines how many pixels the tip has        moved between the acquisitions of images X1 and X1{circumflex        over ( )}.    -   6. Based upon the 2D relationship between images X1 and X2, and        the number of pixels the tip has moved between the acquisitions        of images X1 and X1{circumflex over ( )}, the computer processor        determines the new location of the tip of the tool in image X2.    -   7. The computer processor drives the display to display the tool        tip (or a representation thereof) at its new location with        respect to x-ray image X2.

With reference to FIGS. 25A and 25B, in general, the scope of thepresent invention includes acquiring 3D image data of a skeletalportion, and acquiring first and second 2D x-ray images, from respectivex-ray image views, of the skeletal portion and a portion of a toolconfigured to be advanced into the skeletal portion along a longitudinalinsertion path, while the portion of the tool is disposed at a firstlocation with respect to the insertion path. The location of a portionof the tool with respect to the skeletal portion is identified withinthe first and second 2D x-ray images, by computer processor 22 of system20, by means of image processing, and the computer processor registersthe 2D x-ray images to the 3D image data, e.g., using the techniquesdescribed herein. Thus, a first location of the portion of the tool withrespect to the 3D image data is determined. Subsequently, the tool isadvanced along the longitudinal insertion path with respect to theskeletal portion, such that the portion of the tool is disposed at asecond location along the longitudinal insertion path. Subsequent tomoving the portion of the tool to a second location along the insertionpath, one or more additional 2D x-ray images of at least the portion ofthe tool and the skeletal portion are acquired from a single image view.In accordance with respective applications, the single image view is thesame as one of the original 2D x-ray image views, or is a third,different 2D x-ray image view. Computer processor 22 of system 20identifies the second location of the portion of the tool within the oneor more additional 2D x-ray images, by means of image processing, andderives the second location of the portion of the tool with respect tothe 3D image data, based upon the second location of the portion of thetool within the one or more additional 2D x-ray images, and thedetermined first location of the portion of the tool with respect to the3D image data. Typically, an output is generated in response thereto(e.g., by displaying the derived location of the tool relative to thex-ray image view with respect to which the location has been derived).

In accordance with some applications, first and second 2D x-ray imagesare acquired, from respective x-ray image views, of the skeletal portionand a portion of a tool configured to be advanced into the skeletalportion along a longitudinal insertion path, while the portion of thetool is disposed at a first location with respect to the insertion path.The location of a portion of the tool with respect to the skeletalportion is identified within the first and second 2D x-ray images, bycomputer processor 22 of system 20, by means of image processing, andthe computer processor determines a relationship between the first andsecond 2D x-ray images, e.g., using any one of algorithms 1-4 describedhereinabove. Subsequently, the tool is advanced along the longitudinalinsertion path with respect to the skeletal portion, such that theportion of the tool is disposed at a second location along thelongitudinal insertion path. Subsequent to moving the portion of thetool to the second location along the insertion path, one or moreadditional 2D x-ray images of at least the portion of the tool and theskeletal portion are acquired from a single image view. In accordancewith respective applications, the single image view is the same as oneof the original 2D x-ray image views, or is a third, different 2D x-rayimage view. Computer processor 22 of system 20 identifies the secondlocation of the portion of the tool within the one or more additional 2Dx-ray images by means of image processing, and derives the secondlocation of the portion of the tool with respect to one of the original2D x-ray image views, based upon the second location of the portion ofthe tool that was identified within the additional 2D x-ray image, andthe determined relationship between the first and second 2D x-rayimages. Typically, an output is generated in response thereto (e.g., bydisplaying the derived location of the tool relative to the x-ray imageview with respect to which the location has been derived).

Some examples of the applications of the techniques described withreference to FIGS. 25A and 25B are as follows. For an intervention thatis performed dorsally, initially x-rays may be acquired from lateral andAP views. Subsequent x-rays may be generally acquired from an AP viewonly (with optional periodic checks from the lateral view, as describedhereinabove), with the updated locations of the tool with respect to thelateral view being derived and displayed. It is noted that although, inthis configuration, the C-arm may disturb the intervention, the AP viewprovides the best indication of the location of the tool relative to thespinal cord. Alternatively, subsequent x-rays may be generally acquiredfrom a lateral view only (with optional periodic checks from the AP viewas described hereinabove), with the updated locations of the tool withrespect to the AP view being derived and displayed. Typically, for suchapplications, sets 50 of markers 52 are placed on the patient such thatat least one set of markers is visible from the lateral view. Furtheralternatively, subsequent x-ray may be generally acquired from anoblique view only (with optional periodic checks from the lateral and/orAP view as described hereinabove), with the updated locations of thetool with respect to the AP and/or lateral view being derived anddisplayed. It is noted that the above applications are presented asexamples, and the scope of the present invention includes using thetechniques described with reference to FIGS. 25A and 25B withinterventions that are performed on any portion of the skeletal anatomy,from any direction of approach, and with any type of x-ray image views,mutatis mutandis.

For some applications, the assumption that the tool, after having beeninserted into the vertebra (and typically fixated firmly within thevertebra), has indeed proceeded along an anticipated longitudinalforward path is verified, typically automatically. Consecutive x-rayimages acquired from a same pose are overlaid upon one another to checkwhether, when the images are positioned such that a position of the toolas seen in a second image is longitudinally aligned with a priorposition of the same tool in a first image, the observed anatomies inboth images indeed overlap with one another. Or, alternatively, when theimages are positioned such that the observed anatomies in both imagesoverlap with one another, the position of the tool as seen in a secondimage is indeed longitudinally aligned with a prior position of the sametool in a first image. For some applications, the motion detectionsensor described by the present application is used for verifying thatno motion (or no motion above a certain threshold) of the subject hasoccurred during the acquisition of the subsequent images. For someapplications, comparison of the alignment is manual (visual) by theuser, or automatic (by means of image processing), or any combinationthereof.

Reference is now made to FIGS. 27A-B, which show flowcharts for a methodfor verifying if the tool has indeed proceeded along an anticipatedlongitudinal path, in accordance with some applications of the presentinvention. For some applications, the assumption that once the tool wasinserted into the vertebra (and typically fixated firmly) it has indeedproceeded along an anticipated longitudinal path is verified, typicallyautomatically, as follows:

-   -   1. 3D image data is acquired of the skeletal portion, e.g.,        vertebra (step 334).    -   2. The anticipated longitudinal forward path of the tool is        computed within the 3D image data (step 344) from two x-ray        images (i) acquired (typically using an x-ray imaging device        that is unregistered with respect to the body of the subject)        from different views while the tool is in the same position,        i.e., at a first location along the longitudinal insertion path        of the tool (step 336), (ii) registered with the 3D scan data,        using techniques disclosed by the present application (step        338), and (iii) in each of which a location of the portion of        the tool with respect to the skeletal portion is identified        (step 340), such that the first location of the portion of the        tool is identified with respect to the 3D image data (step 342).    -   3. The tool is moved further, typically forward, to a second        location along the longitudinal insertion path (step 346).    -   4. One or more additional x-ray images is acquired (from any        view, not necessarily from one of the two prior views, see        Algorithm 1 and Algorithm 2 of the 2D-3D registration) (step        348).    -   5. With reference to FIG. 27B, computer processor 22 is used to        facilitate identifying whether the tool has deviated from the        anticipated longitudinal forward path (step 350 of FIG. 27A) as        follows:    -   6. The newly-acquired x-ray image is registered with the 3D scan        data within which the anticipated longitudinal progression,        i.e., forward, path has been computed (step 352).    -   7. The anticipated longitudinal progression path now becomes        registered with the newly-acquired, i.e., additional one or        more, x-ray image; optionally, it may now be shown on the        newly-acquired x-ray image.    -   8. In the newly-acquired x-ray image, the actual tool, and        particularly the distal portion thereof, is identified (step        354) and may be compared with the anticipated longitudinal        progression path to identify whether the tool has deviated from        the anticipated longitudinal forward, e.g., progression, path.        For some applications, the comparison is manual (visual) by the        user, or automatic by the system (typically by means of image        processing), or any combination thereof (It should be noted that        such comparison is typically only in the imaging plane of the        x-ray system.)    -   9. For some applications, if a significant difference (which may        also be defined as above a certain threshold) between the actual        distal portion of the tool and the anticipated longitudinal        progression path has been identified manually (visually) by the        user, or automatically (in pixels, or in absolute distance, by        means of image processing) by the system, then the anticipated        longitudinal progression path may be recalculated by moving the        x-ray source into a substantially different viewing position,        without moving the tool, acquiring another x-ray image, and have        the system recalculate the anticipated longitudinal progression        path using the two most recently acquired x-ray images (i.e.,        the x-ray image just acquired from the substantially different        viewing position and the additional x-ray image acquired in step        348).

Reference is now made to FIGS. 28A-E, which show an example of a toolbending during its insertion, with the bending becoming increasinglyvisible (manually) or identifiable (automatically). (The black line 372is the tool in the x-ray, the solid white line 374 is the anticipatedlongitudinal progression path where it still matches the actual tool,and the dashed white line 376 is where the tool becomes further awayfrom the anticipated longitudinal progression path. Solid white line 374and dashed white line 376 are in some embodiments generated by thesystem. In some embodiments, there is only one white line not brokeninto a solid section and a dashed section.)

For some applications, the image of the tool (a representation thereof,and/or a path thereof) as derived from the 2D images is overlaid uponthe 3D image data of the vertebra as a hologram. As noted hereinabove,since, in accordance with such applications, the images of the vertebraand the tool (or a representation thereof) are input from differentimaging sources, the segmented data of what is the tool (or itsrepresentation) and what is the vertebra is in-built (i.e., it isalready known to the computer processor). For some applications, thecomputer processor utilizes this in-built segmentation to allow theoperator to virtually manipulate the tool with respect to the vertebra,within the hologram. For example, the operator may virtually advance thetool further along its insertion path, or retract the tool and observethe motion of the tool with respect to the vertebra. Or, the computerprocessor may automatically drive the holographic display to virtuallyadvance the tool further along its insertion path, or retract the tool.For some applications, similar techniques are applied to other tools andbodily organs, mutatis mutandis. For example, such techniques could beapplied to a CT image of the heart in combination with 2D angiographicimages of a catheter within the heart.

For some applications, an optical camera is used to acquire opticalimages of a tool. For example, optical camera 114, which is disposed onx-ray C-arm 34, as shown in FIG. 1, may be used. Alternatively oradditionally, an optical camera may be disposed on a separate arm, maybe handheld, may be the back camera of a display such as a tablet ormini-tablet device, may be placed on the surgeon's head, may be placedon another portion of the surgeon's body, and/or may be held by anothermember of the surgical staff. Typically, the computer processor derivesthe location of the tool with respect to the 3D image data, based upon2D images in which the tool was observed and using the registrationtechniques described hereinabove. For some applications, in addition,the computer processor identifies the tool within an optical imageacquired by the optical camera. For some such applications, the computerprocessor then overlays the 3D image data upon the optical image byaligning the location of the tool within the 3D image data and thelocation of the tool within the optical image. The computer processorthen drives an augmented reality display to display the 3D image dataoverlaid upon the optical image. Such a technique may be performed usingany viewing direction of the optical camera within which the tool isvisible, and typically without having to track the position of thesubject with respect to the optical camera.

For some applications, the location of the tool within the optical imagespace is determined by using two or more optical cameras, and/or one ormore 3D optical cameras. For some applications, even with one 2D opticalcamera, the 3D image data is overlaid upon the optical image, byaligning two or more tools from each of the imaging modalities. For someapplications, even with one 2D optical camera and a single tool, the 3Dimage data is overlaid upon the optical image, by acquiring additionalinformation regarding the orientation (e.g., rotation) of the tool,and/or the depth of the tool below the skin. For some applications, suchinformation is derived from 3D image data from which the location of theskin surface relative to the vertebra is derived. Alternatively oradditionally, such information is derived from an x-ray image in whichthe tool and the subject's anatomy are visible. Alternatively oradditionally, such information is derived from the marker set as seen inan x-ray image in which the tool and the subject's anatomy are visible.

As noted hereinabove, since the images of the vertebra and the tool (ora representation thereof) are input from different imaging sources, thesegmented data of what is the tool (or its representation) and what isthe vertebra is in-built (i.e., it is already known to the computerprocessor). For some applications, the computer processor utilizes thisin-built segmentation to allow the operator to virtually manipulate thetool with respect to the vertebra, within an augmented reality display.For example, the operator may virtually advance the tool further alongits insertion path, or retract the tool and observe the motion of thetool with respect to the vertebra. Or, the computer processor mayautomatically drive the augmented reality display to virtually advancethe tool further along its insertion path, or retract the tool.

Although some applications of the present invention have been describedwith reference to 3D CT image data, the scope of the present inventionincludes applying the described techniques to 3D MRI image data. Forsuch applications, 2D projection images (which are geometricallyanalogous to DRRs that are generated from CT images) are typicallygenerated from the MRI image data and are matched to the 2D images,using the techniques described hereinabove. For some applications, othertechniques are used for registering MRI image data to 2D x-ray images.For example, pseudo-CT image data may be generated from the MRI imagedata (e.g., using techniques as described in “Registration of 2D x-rayimages to 3D MRI by generating pseudo-CT data” by van der Bom et al.,Physics in Medicine and Biology, Volume 56, Number 4), and the DRRs thatare generated from the pseudo-CT data may be matched to the x-rayimages, using the techniques described hereinabove.

For some applications, MRI imaging is used during spinal endoscopy, andthe techniques described herein (including any one of the stepsdescribed with respect to FIG. 6) are used to facilitate performance ofthe spinal endoscopy. Spinal endoscopy is an emerging procedure that isused, for example, in spinal decompression. By using an endoscope,typically, tools can be inserted and manipulated via a smaller incisionrelative to current comparable surgery that is used for similarpurposes, such that a smaller entry space provides a larger treatmentspace than in traditional procedures. Typically, such procedures areused for interventions on soft tissue, such as discs. Such tissue istypically visible in MRI images, but is less, or not at all, visible inCT images or in 2D x-ray images. Traditionally, such procedures commencewith needle insertion under C-Arm imaging. A series of dilators are theninserted to gradually broaden the approach path. Eventually, an outertube having a diameter of approximately 1 cm is then kept in place andan endoscope is inserted therethrough. From this point on, the procedureis performed under endoscopic vision.

For some applications, level verification as described hereinabove isapplied to a spinal endoscopy procedure in order to determine thelocation of the vertebra with respect to which the spinal endoscopy isto be performed. Alternatively or additionally, the incision site forthe spinal endoscopy may be determined using bidirectional mapping ofoptical images and x-ray images, as described hereinabove. Alternativelyor additionally, planning of the insertion may be performed upon the 3DMRI data as described hereinabove. Alternatively or additionally, actualinsertion vs. the planned path may be represented upon the 3D MRI dataas described hereinabove. Alternatively or additionally, actualinsertion vs. the planned path may be represented upon a 2D x-ray imageas described hereinabove. For some applications, MRI image data areregistered to intraprocedural 2D x-ray images. Based upon theregistration, additional steps which are generally as describedhereinabove are performed. For example, the needle, dilator, and/orendoscope (and/or a representation thereof, and/or a path thereof) maybe displayed relative to a target within the MRI image data (e.g., a 3DMRI image, a 2D cross-section derived from 3D MRI image data, and/or a2D projection image derived from 3D MRI image data). For someapplications, endoscopic image data are co-registered to intraprocedural2D x-ray images. For example, respective endoscopic image data pointsmay be co-registered with respective locations within theintraprocedural images. For some applications, the co-registeredendoscopic image data are displayed with the intraprocedural images,together with an indication of the co-registration of respectiveendoscopic image data points with respective locations within theintraprocedural images. Alternatively or additionally, endoscopic imagedata are co-registered to MRI image data. For example, respectiveendoscopic image data points may be co-registered with respectivelocations within the MRI image data. For some applications, theco-registered endoscopic image data are displayed with the MRI imagedata, together with an indication of the co-registration of respectiveendoscopic image data points with respective locations within the MRIimage data.

For some applications, the techniques described herein are performed incombination with using a robotic arm, such as a relatively low-costrobotic arm having 5-6 degrees of freedom. In accordance with someapplications, the robotic arm is used for holding, manipulating, and/oractivating a tool, and/or for operating the tool along a pre-programmedpath. For some applications, computer processor 22 drives the roboticarm to perform any one of the aforementioned operations responsively toimaging data, as described hereinabove.

Reference is now made to FIG. 29A, which shows examples of x-ray imagesof an image calibration jig generated by a C-arm that uses an imageintensifier (on the left), and by a C-arm that uses a flat-paneldetector (on the right), such images reflecting prior art techniques.Reference is also made to FIG. 29B, which shows an example of an x-rayimage acquired by a C-arm that uses an image intensifier, the imageincluding a radiopaque component 200 that corresponds to a portion of atool that is known to be straight, and a dotted line 210 overlaid uponthe image indicating how a line (for example, a centerline) defined bythe straight portion would appear if distortions in the image arecorrected, in accordance with some applications of the presentinvention.

As may be observed in the example shown in FIG. 29A, in x-ray imagesgenerated by a C-Arm that uses an image intensifier, there is typicallyimage distortion, which increases toward the periphery of the image. Bycontrast, in images generated using a flat-panel detector, there istypically no distortion. For some applications of the present invention,distortions in x-ray images generated by a C-Arm that uses an imageintensifier are at least partially corrected automatically, by means ofimage processing. For example, the distortion of such images may becorrected in order to then register the corrected image to a 3D imagedata, using the techniques described hereinabove.

Referring to FIG. 29B, for some applications such an x-ray image is atleast partially corrected by computer processor 22 identifying, by meansof image processing, a radiopaque component 200 of an instrument withina portion of the radiographic image. For some applications, theradiopaque component is a portion of the tool that is known to bestraight, a component having a different known shape, and/or two or morefeatures that are disposed in known arrangement with respect to oneanother. For example, the straight shaft of a Jamshidi™ needle may beidentified.

For some applications, in order to at least partially correct an x-rayimage comprising a radiopaque component that is known to be straight,the computer processor uses techniques for automatically identifying acenterline of an object, for example, as described in US 2010-0161022 toTolkowsky, which is incorporated herein by reference, to generate acenterline of said component. Typically, the computer processor then atleast partially corrects the image distortion, in at least a portion ofthe image in which the component that is known to be straight isdisposed, by deforming the portion of the radiographic image, such thatthe centerline of the radiopaque component of the instrument that isknown to be straight appears straight within the radiographic image.FIG. 29B shows an example of how an x-ray image, prior to correcting itsdistortion, comprises component 200 that is known to be straight and yetdoes not appear straight within the image, as can be observed relativeto straight line 210. For some applications, two or more such componentsare identified in respective portions of the image, and distortion ofthose portions of the image are corrected accordingly. For someapplications, distortions in portions of the image in which no suchcomponents are disposed are corrected, based upon distortion correctionparameters that are determined by means of the radiopaque component thatis known to be straight, or known to have a different known shape.

For some applications of the present invention, techniques describedhereinabove are combined with a system that determines the location ofthe tip of a tool with respect to a portion of the subject's body by (a)calculating a location of a proximal portion of the tool that isdisposed outside the subject's body, and (b) based upon the calculatedposition of the proximal portion of the tool, deriving a location of atip of the tool with respect to the portion of the subject's body withrespect to the 3D image data. For example, such techniques may be usedwith a navigation system that, for example, may include the use of oneor more location sensors that are attached to a portion of a tool thatis typically disposed outside the subject's body even during theprocedure. (It is noted that the location sensors that are disposed uponthe tool may be sensors that are tracked by a tracker that is disposedelsewhere, or they may be a tracker that tracks sensors that aredisposed elsewhere, and thereby acts a location sensor of the tool.) Forexample, a tool may be inserted into the subject's vertebra, such thatits distal tip (or a distal portion of the tool) is disposed inside thevertebra, and a location sensor may be disposed on a proximal portion ofthe tool that is disposed outside the subject's body. The navigationsystem typically derives the location of the tip of the tool (or adistal portion of the tool), by detecting the location(s) of thelocation sensor(s) that are disposed on the proximal portion of thetool, and then deriving the location of the tip of the tool (or a distalportion of the tool) based upon an assumed location of the distal tip ofthe tool (or a distal portion of the tool) relative to the locationsensor(s). The navigation system then overlays the derived location ofthe tip of the tip of the tool (or a distal portion of the tool) withrespect to the vertebra upon previously acquired 3D image data (e.g.,images acquired prior to the subject being placed in the operating room,or when the subject was in the operating room, but typically prior tothe commencement of the intervention). Alternatively or additionally,the location of a proximal portion of the tool that is disposed outsidethe subject's body may be calculated by video tracking the proximalportion of the tool, and/or by means of tracking motion of a portion ofa robot to which the proximal portion of the tool is coupled, relativeto a prior known position, e.g., based upon the values of the joints ofthe robot relative to the corresponding values of the joints of therobot at the prior known position.

In such cases, there may be errors associated with determining thelocation of the tip of the tool (or a distal portion of the tool), basedupon the assumed location of the distal tip of the tool (or a distalportion of the tool) relative to the location sensor(s) being erroneous,e.g., due to slight bending of the tool upon being inserted into thevertebra. Therefore, for some applications, during the procedure,typically periodically, 2D x-ray images are acquired within which theactual tip of tool (or distal portion of the tool) within the vertebrais visible. The location of the tip of the tool (or distal portion ofthe tool) with respect to the vertebra as observed in the 2D x-rayimages is determined with respect to the 3D image data, by registeringthe 2D x-ray images to the 3D image data. For example, the 2D x-rayimages may be registered to the 3D image data using techniques describedhereinabove. In this manner, the actual location of the tip of the tool(or distal portion of the tool) with respect to the vertebra isdetermined with respect to the 3D image data. For some applications, inresponse thereto, errors in the determination of the location of the tipof the tool (or distal portion of the tool) with respect to the vertebrawithin the 3D image space resulting from the navigation system, areperiodically corrected by system 20. For example, based upon thedetermined location of at least the tip of the tool (or distal portionof the tool), the computer processor may drive the display to update theindication of the location of the tip of the tool (or distal portion ofthe tool) with respect to the vertebra with respect to the 3D imagedata. For some applications, the navigation systems comprise the use ofaugmented reality, or virtual reality, or robotic manipulation of tools,or any combination thereof.

By way of illustration and not limitation, it is noted that the scope ofthe present invention includes applying the apparatus and methodsdescribed herein to any one of the following applications:

-   -   Multiple tool insertions (e.g., towards both pedicles) in the        same vertebra.    -   Any type of medical tool or implant, including, Jamshidi™        needles, k-wires, pedicle markers, screws, endoscopes, RF        probes, laser probes, injection needles, etc.    -   An intervention that is performed from a lateral approach, in        which case the functional roles of the AP and lateral x-ray        views described hereinabove are typically switched with one        another.    -   Interventions using x-ray views other than lateral and AP views        as an alternative or in addition to such views. For example,        oblique imaging views may be used.    -   An intervention that is performed from an anterior, oblique        and/or posterior interventional approach.    -   Interventions performed upon multiple vertebrae. Even for such        cases, the intra-operative x-ray images of the vertebrae are        typically registered with the 3D image data of the corresponding        vertebrae on an individual basis.    -   Interventions performed on discs in between vertebrae.    -   Interventions performed on nerves.    -   Tool insertion under x-ray in a video imaging mode.    -   Use of certain features of system 20 utilizing intraprocedural        2D x-ray imaging, but without utilizing preprocedural 3D        imaging.    -   Use of certain features of system 20 without some or all of the        above-described disposable items, such as a drape.    -   Various orthopedic surgeries, such as surgeries performed on        limbs and/or joints.    -   Interventions in other body organs.

For some applications system 20 includes additional functionalities tothose described hereinabove. For example, the computer processor maygenerate an output that is indicative of a current level of accuracy(e.g., of verification of the vertebral level, determination of theinsertion site, and/or registration of the 3D image data to the 2Dimages), e.g., based upon a statistical calculation of the possibleerror. For some applications, the computer processor generates a promptindicating that a new x-ray from one or more views should be acquired.For example, the computer processor may generate such a prompt based onthe time elapsed since a previous x-ray acquisition from a given view,and/or based on the distance a tool has moved since a previous x-rayacquisition from a given view, and/or based on observed changes in theposition of markers 52 relative to the C-arm.

Applications of the invention described herein can take the form of acomputer program product accessible from a computer-usable orcomputer-readable medium (e.g., a non-transitory computer-readablemedium) providing program code for use by or in connection with acomputer or any instruction execution system, such as computer processor22. For the purpose of this description, a computer-usable or computerreadable medium can be any apparatus that can comprise, store,communicate, propagate, or transport the program for use by or inconnection with the instruction execution system, apparatus, or device.The medium can be an electronic, magnetic, optical, electromagnetic,infrared, or semiconductor system (or apparatus or device) or apropagation medium. Typically, the computer-usable or computer readablemedium is a non-transitory computer-usable or computer readable medium.

Examples of a computer-readable medium include a semiconductor or solidstate memory, magnetic tape, a removable computer diskette, arandom-access memory (RAM), a read-only memory (ROM), a rigid magneticdisk and an optical disk. Current examples of optical disks includecompact disk-read only memory (CD-ROM), compact disk-read/write (CD-R/W)and DVD. For some applications, cloud storage, and/or storage in aremote server is used.

A data processing system suitable for storing and/or executing programcode will include at least one processor (e.g., computer processor 22)coupled directly or indirectly to memory elements (such as memory 24)through a system bus. The memory elements can include local memoryemployed during actual execution of the program code, bulk storage, andcache memories which provide temporary storage of at least some programcode in order to reduce the number of times code must be retrieved frombulk storage during execution. The system can read the inventiveinstructions on the program storage devices and follow theseinstructions to execute the methodology of the embodiments of theinvention.

Network adapters may be coupled to the processor to enable the processorto become coupled to other processors or remote printers or storagedevices through intervening private or public networks. Modems, cablemodem and Ethernet cards are just a few of the currently available typesof network adapters.

Computer program code for carrying out operations of the presentinvention may be written in any combination of one or more programminglanguages, including an object-oriented programming language such asJava, Smalltalk, C++ or the like and conventional procedural programminglanguages, such as the C programming language or similar programminglanguages.

It will be understood that blocks of the flowchart shown in FIGS. 7,14A, and 14B, combinations of blocks in the flowcharts, as well as anyone of the algorithms described herein, can be implemented by computerprogram instructions. These computer program instructions may beprovided to a processor of a general-purpose computer, special purposecomputer, or other programmable data processing apparatus to produce amachine, such that the instructions, which execute via the processor ofthe computer (e.g., computer processor 22) or other programmable dataprocessing apparatus, create means for implementing the functions/actsspecified in the flowcharts and/or algorithms described in the presentapplication. These computer program instructions may also be stored in acomputer-readable medium (e.g., a non-transitory computer-readablemedium) that can direct a computer or other programmable data processingapparatus to function in a particular manner, such that the instructionsstored in the computer-readable medium produce an article of manufactureincluding instruction means which implement the function/act specifiedin the flowchart blocks and algorithms. The computer programinstructions may also be loaded onto a computer or other programmabledata processing apparatus to cause a series of operational steps to beperformed on the computer or other programmable apparatus to produce acomputer implemented process such that the instructions which execute onthe computer or other programmable apparatus provide processes forimplementing the functions/acts specified in the flowcharts and/oralgorithms described in the present application.

Computer processor 22 and the other computer processors described hereinare typically hardware devices programmed with computer programinstructions to produce a special purpose computer. For example, whenprogrammed to perform the algorithms described herein, the computerprocessor typically acts as a special purpose skeletal-surgery-assistingcomputer processor. Typically, the operations described herein that areperformed by computer processors transform the physical state of amemory, which is a real physical article, to have a different magneticpolarity, electrical charge, or the like depending on the technology ofthe memory that is used.

It will be appreciated by persons skilled in the art that the presentinvention is not limited to what has been particularly shown anddescribed hereinabove. Rather, the scope of the present inventionincludes both combinations and subcombinations of the various featuresdescribed hereinabove, as well as variations and modifications thereofthat are not in the prior art, which would occur to persons skilled inthe art upon reading the foregoing description.

The invention claimed is:
 1. Apparatus for registering a 2D radiographicimage of a targeted skeletal portion within a body of a patient to 3Dimage data of the targeted skeletal portion of the same patient, and foruse with: (a) a 3D imaging device configured to acquire 3D image data ofthe targeted skeletal portion of the patient, and (b) a 2D radiographicimaging device that is unregistered with respect to the patient's bodyand configured to acquire a 2D radiographic image of the targetedskeletal portion of the patient, the apparatus comprising: (i) amachine-learning engine configured to generate machine-learning databased on: the 3D image data of the targeted skeletal portion of thepatient, a database of 2D projection images generated from the 3D imagedata of the targeted skeletal portion of the patient, and respectivevalues of one or more viewing parameters corresponding to each 2Dprojection image; and (ii) at least one computer processor configuredto: receive the machine-learning data, receive the 2D radiographic imageof the targeted skeletal portion of the patient, and register the 2Dradiographic image of the targeted skeletal portion of the patient tothe 3D image data of the targeted skeletal portion of the patient byusing the machine-learning data to find a 2D projection from the 3Dimage data of the targeted skeletal portion of the patient that matchesthe 2D radiographic image of the targeted skeletal portion of thepatient.
 2. The apparatus according to claim 1, wherein the targetedskeletal portion is a targeted vertebra of a spine of the patient. 3.The apparatus according to claim 1, wherein the respective values of theone or more viewing parameters comprise, respectively, values of aviewing distance and a viewing angle corresponding to each 2D projectionimage.
 4. The apparatus according to claim 1, wherein given a particular2D projection image, the machine-learning engine is configured to learnto suggest simulated respective values of viewing parameters thatcorrespond to that 2D projection image.
 5. The apparatus according toclaim 1, wherein the 2D radiographic imaging device is configured toacquire an intraoperative 2D radiographic image of the targeted skeletalportion of the patient, and the at least one computer processor isconfigured to (a) receive the intraoperative 2D radiographic image ofthe targeted skeletal portion of the patient, and (b) register theintraoperative 2D radiographic image to the 3D image data of thetargeted skeletal portion of the patient by using the machine-learningdata to find a 2D projection from the 3D image data of the targetedskeletal portion of the patient that matches the intraoperative 2Dradiographic image of the targeted skeletal portion of the patient. 6.The apparatus according to claim 1, wherein the at least one computerprocessor is configured to use the machine-learning data to find a 2Dprojection from the 3D image data of the targeted skeletal portion ofthe patient that matches the 2D radiographic image of the targetedskeletal portion of the patient by: using the obtained machine-learningdata to limit a search space in which a 2D projection from the 3D imagedata of the targeted skeletal portion of the patient that matches the 2Dradiographic image of the targeted skeletal portion of the patientshould be searched for; and searching for a 2D projection that matchesthe 2D radiographic image of the targeted skeletal portion of thepatient only within the limited search space.
 7. The apparatus accordingto claim 1, wherein the at least one computer processor is furtherconfigured to: identify a feature of the targeted skeletal portion ofthe patient within the 3D image data of the targeted skeletal portion ofthe patient; and determine a location of the feature with respect to the2D radiographic image of the targeted skeletal portion of the patient bydetermining the location of the feature within the 2D projection imagethat matches the 2D radiographic image of the targeted skeletal portionof the patient, the location of the feature within the 2D projectionimage corresponding to the location of the feature within the matching2D radiographic image of the targeted skeletal portion of the patient.8. The apparatus according to claim 7, wherein the at least one computerprocessor is configured to determine the location of the feature withinthe 2D projection image that matches the 2D radiographic image of thetargeted skeletal portion of the patient by generating a line extendingfrom (a) a simulated camera corresponding to the 2D projection thatmatches the 2D radiographic image of the targeted skeletal portion ofthe patient to (b) the feature of the skeletal portion within the 3Dimage data of the targeted skeletal portion of the patient, and inresponse thereto determining the location of the feature with respect tothe 2D radiographic image of the targeted skeletal portion of thepatient.
 9. The apparatus according to claim 1, wherein: (a) theapparatus is for use with a tool configured to be advanced into askeletal portion within the body of the patient, (b) the 2D radiographicimage of the targeted skeletal portion of the patient is a first 2Dradiographic image of the targeted skeletal portion of the patient, (c)the 2D radiographic imaging device is configured to acquire a second 2Dradiographic image of the targeted skeletal portion of the patient, and(d) the at least one computer processor is configured to (i) registerthe second 2D radiographic image of the targeted skeletal portion of thepatient to the 3D image data of the targeted skeletal portion of thepatient, and (ii) identify a portion of the tool in each of the firstand second 2D radiographic images of the targeted skeletal portion ofthe patient by means of image processing.
 10. The apparatus according toclaim 9, wherein the portion of the tool is a portion of the toolselected from the group consisting of: a tip of the tool, and a shaft ofthe tool.
 11. The apparatus according to claim 9, wherein the at leastone computer processor is configured to register the second 2Dradiographic image of the targeted skeletal portion of the patient tothe 3D image data of the targeted skeletal portion of the patient byusing the machine-learning data to find a 2D projection from the 3Dimage data of the targeted skeletal portion of the patient that matchesthe second 2D radiographic image of the targeted skeletal portion of thepatient.
 12. The apparatus according to claim 9, wherein the at leastone computer processor is configured to, based on the identified portionof the tool in each of the first and second 2D radiographic images ofthe targeted skeletal portion of the patient, determine a location ofthe portion of the tool with respect to the 3D image data of thetargeted skeletal portion of the patient.
 13. The apparatus according toclaim 12, wherein the apparatus is for use with a display, and whereinthe at least one computer processor is configured to drive the displayto display the location of the portion of the tool with respect to the3D image data of the targeted skeletal portion of the patient.
 14. Theapparatus according to claim 12, wherein the apparatus is for use with arobot, and wherein the at least one computer processor is configured todrive the robot to manipulate, place, or activate the tool in responseto the determined location of the portion of the tool with respect tothe 3D image data of the targeted skeletal portion of the patient. 15.The apparatus according to claim 12, wherein the at least one computerprocessor is configured to: derive a relationship between the locationof the portion of the tool with respect to the 3D image data of thetargeted skeletal portion of the patient and a given location within the3D image data of the targeted skeletal portion of the patient; andgenerate an output that is indicative of the relationship between thelocation of the portion of the tool with respect to the 3D image data ofthe targeted skeletal portion of the patient and the given locationwithin the 3D image data of the targeted skeletal portion of thepatient.
 16. The apparatus according to claim 15, wherein the computerprocessor is configured to generate the output that is indicative of therelationship between the location of the portion of the tool withrespect to the 3D image data of the targeted skeletal portion of thepatient and the given location within the 3D image data of the targetedskeletal portion of the patient upon an image selected from the groupconsisting of: a 2D cross-section of the targeted skeletal portion ofthe patient that is derived from the 3D image data of the targetedskeletal portion of the patient, a 2D projection of the targetedskeletal portion of the patient that is derived from the 3D image dataof the targeted skeletal portion of the patient, a 3D image of thetargeted skeletal portion of the patient that is derived from the 3Dimage data of the targeted skeletal portion of the patient, a 2Dradiographic image of the targeted skeletal portion of the patient thatis registered with the 3D image data of the targeted skeletal portion ofthe patient, and at least one of the first and second 2D radiographicimages of the targeted skeletal portion of the patient.
 17. Theapparatus according to claim 12, wherein the apparatus is for use with anavigation system, and wherein the at least one computer processor isconfigured to: (A) use the navigation system to derive a location of theportion of the tool with respect to the patient's body, with respect tothe 3D image data of the targeted skeletal portion of the patient, by:calculating a location of a proximal portion of the tool while theproximal portion of the tool is disposed outside the patient's body, andbased upon the calculated location of the proximal portion of the tool,deriving a location of a distal portion of the tool with respect to thepatient's body, with respect to the 3D image data of the targetedskeletal portion of the patient, and (B) based on the determinedlocation of the portion of the tool with respect to the 3D image data ofthe targeted skeletal portion of the patient, determine a relationshipbetween (i) the derived location of the portion of the tool with respectto the patient's body, with respect to the 3D image data of the targetedskeletal portion of the patient, and (ii) the determined location of theportion of the tool with respect to the 3D image data of the targetedskeletal portion of the patient.
 18. The apparatus according to claim17, wherein the at least one computer processor is configured to, basedon the determined relationship, update the derived location of thedistal portion of the tool with respect to the portion of the patient'sbody, with respect to the 3D image data of the targeted skeletal portionof the patient.
 19. The apparatus according to claim 17, wherein thecomputer processor is configured to use the navigation system tocalculate the location of the proximal portion of the tool by using atechnique selected from the group consisting of: (a) detecting alocation of at least one location sensor that is disposed on theproximal portion of the tool, (b) video tracking the proximal portion ofthe tool, and (c) tracking motion of a portion of a robot to which theproximal portion of the tool is coupled.
 20. The apparatus according toclaim 17, wherein the apparatus is for use with a robot, and wherein theat least one computer processor is configured to drive the robot tomanipulate, place, or activate the tool in response to the determinedlocation of the portion of the tool with respect to the 3D image data ofthe targeted skeletal portion of the patient.